AI agents find $4.6M in blockchain smart contract exploits

From the article:

We introduce SCONE-bench—the first benchmark that evaluates agents’ ability to exploit smart contracts, measured by the total dollar value[2] of simulated stolen funds. For each target contract(s), the agent is prompted to identify a vulnerability and produce an exploit script that takes advantage of the vulnerability so that, when executed, the executor’s native token balance increases by a minimum threshold. Instead of relying on bug bounty or speculative models, SCONE-bench uses on-chain assets to directly quantify losses.

...

First, we evaluated 10 models[3] across all 405 benchmark problems. Collectively, these models produced turnkey exploits for 207 (51.11%) of these problems, yielding $550.1 million in simulated stolen funds.[4]

Second, to control for potential data contamination, we evaluated the same 10 models on 34 problems that were exploited after March 1, 2025 (these models’ latest knowledge cutoff). Collectively, Opus 4.5, Sonnet 4.5, and GPT-5 produced exploits for 19 of these problems (55.8%), yielding a maximum of $4.6 million in simulated stolen funds.[5] The top performing model, Opus 4.5, successfully exploited 17 of these problems (50%), corresponding to $4.5 million in simulated stolen funds—an estimate of how much these AI agents could have stolen had they been pointed to these smart contracts throughout 2025.[6]

Third, to assess our agent’s ability to uncover completely novel zero-day exploits, we evaluated the Sonnet 4.5 and GPT-5 agents on October 3, 2025 against 2,849 recently deployed contracts that contained no known vulnerabilities. The agents both uncovered two novel zero-day vulnerabilities and produced exploits worth $3,694,[7] with GPT-5 doing so at an API cost of $3,476, demonstrating as a proof-of-concept that profitable, real-world autonomous exploitation is technically feasible.

...

Over the last year, frontier models' exploit revenue on the 2025 problems doubled roughly every 1.3 months (Figure 1). We attribute the increase in total exploit revenue to improvements in agentic capabilities like tool use, error recovery, and long-horizon task execution. Even though we expect this doubling trend to plateau eventually, it remains a striking demonstration of how fast exploit revenue increased based on capability improvements in just a year.

For a new vulnerability they discovered:

We found no way to contact the developer, a common issue due to the anonymous nature of blockchains. Four days after our agent’s discovery, a real attacker independently exploited the same flaw and drained approximately $1,000 worth of fees.

Matt Levine has comments:

I love “produced exploits worth $3,694 … at an API cost of $3,476.” That is: It costs money to make a superintelligent computer think; the more deeply it thinks, the more money it costs. There is some efficient frontier: If the computer has to think $10,000 worth of thoughts to steal $5,000 worth of crypto, it’s not worth it. Here, charmingly, the computer thought just deeply enough to steal more money than its compute costs. For one thing, that suggests that there are other crypto exploits that are too complicated for this research project, but that a more intense AI effort could find.

For another thing, it feels like just a pleasing bit of self-awareness on the AI’s part. Who among us has not sat down to some task thinking “this will be quick and useful,” only to find out that it took twice as long as we expected and accomplished nothing? Or put off some task thinking it would be laborious and useless, only to eventually do it quickly with great results? The AI hit the efficient frontier exactly; nice work!

But it seems like anyone writing smart contracts ought to be setting up automatic AI security reviews as part of their release process, and that would certainly make more money for the AI firms.