It was a nice, short read. I didn't read too much of the attached article on Anthropic's website that goes into the technical details, but from what I read on the Mozilla blog post, I think this...
It was a nice, short read. I didn't read too much of the attached article on Anthropic's website that goes into the technical details, but from what I read on the Mozilla blog post, I think this is pretty cool. I don't want AI to take over the entirety of software development, but this is one of those instances where I think it's being used well, especially because test cases were developed to prove that the AI wasn't just hallucinating the bugs.
I ultimately feel like there will be no option but to take this AI-assisted approach going forward, because inevtiably hackers or other "bad guys" can do the same to discover bugs to use as exploits.
I have to say, Anthropic has gained quite a bit of respect from me recently. Obviously not everything they do is good (and frankly the bar is quite low considering the comically evil OpenAI is their competitor), but this is the kind of AI applications that I think really do more good than harm.
The sad reality is that virtually no big company is going to be able take a moral stand against selling to the military (and subsequently following orders) for long. The pile of money on the table...
The sad reality is that virtually no big company is going to be able take a moral stand against selling to the military (and subsequently following orders) for long. The pile of money on the table is too big.
Anthropic’s team got in touch with Firefox engineers after using Claude to identify security bugs in our JavaScript engine. Critically, their bug reports included minimal test cases that allowed our security team to quickly verify and reproduce each issue.
Within hours, our platform engineers began landing fixes, and we kicked off a tight collaboration with Anthropic to apply the same technique across the rest of the browser codebase. In total, we discovered 14 high-severity bugs and issued 22 CVEs as a result of this work. All of these bugs are now fixed in the latest version of the browser.
In addition to the 22 security-sensitive bugs, Anthropic discovered 90 other bugs, most of which are now fixed. A number of the lower-severity findings were assertion failures, which overlapped with issues traditionally found through fuzzing, an automated testing technique that feeds software huge numbers of unexpected inputs to trigger crashes and bugs. However, the model also identified distinct classes of logic errors that fuzzers had not previously uncovered.
[...]
The scale of findings reflects the power of combining rigorous engineering with new analysis tools for continuous improvement. We view this as clear evidence that large-scale, AI-assisted analysis is a powerful new addition in security engineers’ toolbox. Firefox has undergone some of the most extensive fuzzing, static analysis, and regular security review over decades. Despite this, the model was able to reveal many previously unknown bugs. This is analogous to the early days of fuzzing; there is likely a substantial backlog of now-discoverable bugs across widely deployed software.
Speaking from experience, using Claude for code audits is really useful. It can be a bit time consuming as it is prone to exaggeration and error. My process has improved quite a bit over time....
Speaking from experience, using Claude for code audits is really useful. It can be a bit time consuming as it is prone to exaggeration and error. My process has improved quite a bit over time.
Looking at an early code audit I did on a fairly large existing project that was not designed with AI, it surfaced about 60 purported issues. The prompting was pretty simplistic, basically asking for a code audit to look for security issues, duplicative code, bad code smells, useful refactoring, and to report them as GitHub issues.
On a second pass through, about 40% were outright fabrications. Things like “error reporting uses inconsistent message keys, like error, errors, or message. Should make consistent throughout codebase” when in fact errors were only ever reported using an array of errors keyed as “errors”.
After filtering the absolute nonsense, the remainder were about 10% actually perfectly correct issues that went directly to useful fixes. The other 50% contained a mixture of truth and fiction. Largely valuable, but took a significant amount of time to separate out what was real.
Now my process is quite a lot better, but it consumes a lot of tokens. I now ask it for four passes:
an initial pass to go into GitHub issues
a detailed report showing exact code lines where the purported bugs are and test cases where applicable that currently fail, cancelling any issues that are found to be incorrect
a pull request that resolves the issue, without breaking any tests or linting rules
an analysis of the pull request, asking whether these changes are necessary, whether they are reinventing the wheel when a common well tested library could be used instead
It was a nice, short read. I didn't read too much of the attached article on Anthropic's website that goes into the technical details, but from what I read on the Mozilla blog post, I think this is pretty cool. I don't want AI to take over the entirety of software development, but this is one of those instances where I think it's being used well, especially because test cases were developed to prove that the AI wasn't just hallucinating the bugs.
I ultimately feel like there will be no option but to take this AI-assisted approach going forward, because inevtiably hackers or other "bad guys" can do the same to discover bugs to use as exploits.
I have to say, Anthropic has gained quite a bit of respect from me recently. Obviously not everything they do is good (and frankly the bar is quite low considering the comically evil OpenAI is their competitor), but this is the kind of AI applications that I think really do more good than harm.
The sad reality is that virtually no big company is going to be able take a moral stand against selling to the military (and subsequently following orders) for long. The pile of money on the table is too big.
From the article:
[...]
Speaking from experience, using Claude for code audits is really useful. It can be a bit time consuming as it is prone to exaggeration and error. My process has improved quite a bit over time.
Looking at an early code audit I did on a fairly large existing project that was not designed with AI, it surfaced about 60 purported issues. The prompting was pretty simplistic, basically asking for a code audit to look for security issues, duplicative code, bad code smells, useful refactoring, and to report them as GitHub issues.
On a second pass through, about 40% were outright fabrications. Things like “error reporting uses inconsistent message keys, like error, errors, or message. Should make consistent throughout codebase” when in fact errors were only ever reported using an array of errors keyed as “errors”.
After filtering the absolute nonsense, the remainder were about 10% actually perfectly correct issues that went directly to useful fixes. The other 50% contained a mixture of truth and fiction. Largely valuable, but took a significant amount of time to separate out what was real.
Now my process is quite a lot better, but it consumes a lot of tokens. I now ask it for four passes:
And then manual review for what’s left.