When provided with CVE descriptions of 15 different vulnerabilities and a set of tools useful for exploitation, GPT-4 was capable of autonomously exploiting 13 of which, yielding an 87% success rate
Potentially interesting, but potentially also not interesting at all. I didn't read the entire paper so I might have missed it while skimming. I don't see training data and cutoff dates for the...
Potentially interesting, but potentially also not interesting at all. I didn't read the entire paper so I might have missed it while skimming. I don't see training data and cutoff dates for the LLMs being taken into account. Similarly, I don't see them taking into account how old the CVEs are, how much information is available on the internet about exploiting them, etc.
Looking at paragraph 5.3 where they remove the CVE descriptions, I get a strong sense that part of the success is how much information about the CVEs was present in the set of training data for the models.
This is a generic model - so it would be easy to fine tune an even more competent one. It sounds like this has the potential to massively amplify the capabilities of your average skript kiddy, and...
I get a strong sense that part of the success is how much information about the CVEs was present in the set of training data for the models.
This is a generic model - so it would be easy to fine tune an even more competent one. It sounds like this has the potential to massively amplify the capabilities of your average skript kiddy, and who knows what it could do for state/corporate actors.
That's not really what I was trying to convey though. I am saying that it isn't all that surprising that a LLM can get pretty far in this area as long as the training data contains enough...
That's not really what I was trying to convey though. I am saying that it isn't all that surprising that a LLM can get pretty far in this area as long as the training data contains enough information about the exploits. This being a generic model (multiple actually if you read the paper), is probably what actually allows it to be successful. Because the internet is full of information about CVEs and how to exploit them. So this is only news worthy, imho, if a LLM managed to do this on CVEs beyond their training data cutoff.
It isn't all that far removed from other technical tasks they are already being used in.
So when you say that they amplify the capabilities of your average script kiddy. Yes, but that was already the case and this paper potentially doesn't point out anything new.
and who knows what it could do for state/corporate actors.
Not much? I highly recommend you at least check out paragraph 5.3 from the paper :)
I read the paper in full -- take a closer look. ...
I read the paper in full -- take a closer look.
For GPT-4, the knowledge cutoff date was November 6th, 2023. Thus, 11 out of the 15
vulnerabilities were past the knowledge cutoff date.
...
We further note that GPT-4 achieves an 82% success rate when only considering vulnerabili-
ties after the knowledge cutoff date (9 out of 11 vulnerabilities).
This was my first and most immediate thought. Age of the CVE and complications of actually developing a viable PoC/exploit kit is certainly one thing to be concerned about but if LLMs with...
has the potential to massively amplify the capabilities of your the skript kiddy
This was my first and most immediate thought. Age of the CVE and complications of actually developing a viable PoC/exploit kit is certainly one thing to be concerned about but if LLMs with specific training sets continue to evolve (and I would be darn sure APTs are already doing so) that just creates more low hanging fruit for the dirtbag bag guys and more headaches for security engineers.
On the flip side - it would be really great if other LLMs were also leveraged by the good guys to create useful IoCs and even patches for some of these CVEs and exploits created.
I could see this becoming something akin to an arms race and it makes me think there will be a lot more regulation in the LLMs and AI space (hopefully) but I also feel like Pandora has opened the box and there is not putting the genie back in the bottle.
I agree that the cutoff time needs to be taken into account, but contrary to your comment's claim it is in fact mentioned in the paper multiple times. According to the paper 11 out of the 15...
I agree that the cutoff time needs to be taken into account, but contrary to your comment's claim it is in fact mentioned in the paper multiple times. According to the paper 11 out of the 15 tested CVEs were past the cutoff date. It appears that both unsuccessful cases were in fact CVEs released after the cutoff date, but that still leaves 9 out of 11 successful cases. (See some relevant quotes from the paper below.)
Characteristics of the vulnerabilities. Our vulnerabilities span website vulnerabilities, container vulnerabilities, and vulnerable Python packages. Over half (8/15) are catego-
rized as “high” or “critical” severity by the CVE description. Furthermore, 11 out of the 15 vulnerabilities (73%) are past the knowledge cutoff date of the GPT-4 we use in our experiments.
For GPT-4, the knowledge cutoff date was November 6th, 2023. Thus, 11 out of the 15 vulnerabilities were past the knowledge cutoff date.
We further note that GPT-4 achieves an 82% success rate when only considering vulnerabilities after the knowledge cutoff date (9 out of 11 vulnerabilities).
After removing the CVE description, the success rate falls from 87% to 7%. This suggests that determining the vulnerability is extremely challenging. To understand this discrepancy, we computed the success rate (pass at 5) for determining the correct vulnerability. Surprisingly, GPT-4 was able to identify the correct vulnerability 33.3% of the time. Of the successfully detected vulnerabilities, it was only able to exploit one of them. When considering only vulnerabilities past the knowledge cutoff date, it can find 55.6% of them.
Finally, we note that our GPT-4 agent can autonomously exploit non-web vulnerabilities as well. For example, consider the Astrophy RCE exploit (CVE-2023-41334). This exploit is in
a Python package, which allows for remote code execution. Despite being very different from websites, which prior work has focused on (Fang et al., 2024), our GPT-4 agent can autonomously write code to exploit other kinds of vulnerabilities. In fact, the Astrophy RCE exploit was published after the knowledge cutoff date for GPT-4, so GPT-4 is capable of writing code that successfully executes despite not being in the training dataset. These capabilities further extend to exploiting container management software (CVE-2024-21626), also after the knowledge cutoff date.
That being said I do believe that any generalized conclusion being drawn from this paper should be taken with a grain of salt given the very small overall sample size of CVEs tested. Also it would...
That being said I do believe that any generalized conclusion being drawn from this paper should be taken with a grain of salt given the very small overall sample size of CVEs tested.
Also it would be interesting to see the actual exploits GPT4 came up with to analyse the actual approaches taken. Unfortunately I haven't been able to determine whether any supplementary material of the like has been published by the authors (yet). They do state that they want to withhold the exact prompts used in their experiments for ethical reasons and will only provide them upon request, but I don't believe that argument applies to the final exploit code, given that the paper only deals with fully, publicly disclosed CVEs.
Finally keep in mind, that the paper I linked is a (very recent) preprint. So peer-review is most likely still pending.
Potentially interesting, but potentially also not interesting at all. I didn't read the entire paper so I might have missed it while skimming. I don't see training data and cutoff dates for the LLMs being taken into account. Similarly, I don't see them taking into account how old the CVEs are, how much information is available on the internet about exploiting them, etc.
Looking at paragraph 5.3 where they remove the CVE descriptions, I get a strong sense that part of the success is how much information about the CVEs was present in the set of training data for the models.
This is a generic model - so it would be easy to fine tune an even more competent one. It sounds like this has the potential to massively amplify the capabilities of your average skript kiddy, and who knows what it could do for state/corporate actors.
That's not really what I was trying to convey though. I am saying that it isn't all that surprising that a LLM can get pretty far in this area as long as the training data contains enough information about the exploits. This being a generic model (multiple actually if you read the paper), is probably what actually allows it to be successful. Because the internet is full of information about CVEs and how to exploit them. So this is only news worthy, imho, if a LLM managed to do this on CVEs beyond their training data cutoff.
It isn't all that far removed from other technical tasks they are already being used in.
So when you say that they amplify the capabilities of your average script kiddy. Yes, but that was already the case and this paper potentially doesn't point out anything new.
Not much? I highly recommend you at least check out paragraph 5.3 from the paper :)
I read the paper in full -- take a closer look.
...
This was my first and most immediate thought. Age of the CVE and complications of actually developing a viable PoC/exploit kit is certainly one thing to be concerned about but if LLMs with specific training sets continue to evolve (and I would be darn sure APTs are already doing so) that just creates more low hanging fruit for the dirtbag bag guys and more headaches for security engineers.
On the flip side - it would be really great if other LLMs were also leveraged by the good guys to create useful IoCs and even patches for some of these CVEs and exploits created.
I could see this becoming something akin to an arms race and it makes me think there will be a lot more regulation in the LLMs and AI space (hopefully) but I also feel like Pandora has opened the box and there is not putting the genie back in the bottle.
I agree that the cutoff time needs to be taken into account, but contrary to your comment's claim it is in fact mentioned in the paper multiple times. According to the paper 11 out of the 15 tested CVEs were past the cutoff date. It appears that both unsuccessful cases were in fact CVEs released after the cutoff date, but that still leaves 9 out of 11 successful cases. (See some relevant quotes from the paper below.)
That being said I do believe that any generalized conclusion being drawn from this paper should be taken with a grain of salt given the very small overall sample size of CVEs tested.
Also it would be interesting to see the actual exploits GPT4 came up with to analyse the actual approaches taken. Unfortunately I haven't been able to determine whether any supplementary material of the like has been published by the authors (yet). They do state that they want to withhold the exact prompts used in their experiments for ethical reasons and will only provide them upon request, but I don't believe that argument applies to the final exploit code, given that the paper only deals with fully, publicly disclosed CVEs.
Finally keep in mind, that the paper I linked is a (very recent) preprint. So peer-review is most likely still pending.
Yeah basically, it might've gotten that description, but it probably knew much more about prior to being given the description.