Here is the abstract: From a Facebook post by one of the authors, Suvrat Raju:
Here is the abstract:
We present a suite of 50 unpublished research-level theoretical physics problems with verifiable answers. Without AI assistance, a typical problem in our collection could be solved by a world expert in about a day or less but would take considerably longer for experts in adjacent specialties. We test the ability of off-the-shelf AI agents to tackle these problems autonomously, without human hints. When given access to tools available in a realistic setting, including the ability to search the literature and write scripts for numerical and symbolic work, the best-performing agent, GPT-6-Astra harnessed in Codex, obtains the final answers with a near-perfect accuracy of 98% at an average cost under USD 3 per problem. On average, agents show only small changes in accuracy when denied web search and access to background research papers, suggesting significant memorization of the literature. These results suggest that state-of-the-art agents can accurately perform concrete calculations required across theoretical physics research. We comment on the broader implications of these findings and discuss possible explanations for the current lack of agent-led breakthroughs in theoretical physics despite their outstanding problem-solving abilities.
From a Facebook post by one of the authors, Suvrat Raju:
About 6 months ago, just for fun, I prompted a frontier agent with an unpublished problem whose answer I had worked out. The agent got the answer wrong but it was impressive that it managed to make any progress at all, and could solve the problem after a few hints.
Praneeth suggested it might be fun to collect a suite of such problems along the lines of the mathematicians' First Proof assessment. Soon we had a small core group and started requesting experts across the world to contribute problems. Problem collection was a slow process, since we were making a significant ask of contributors. We wanted unpublished problems, with clean verifiable answers. So, our contributors had to either sacrifice potential publication elsewhere, or else undertake a small research project just for us.
Even as problems started coming in, the agents started improving as newer models were released. We kept urging people to push up the difficulty of their problems. But, there is a ceiling to this difficulty since even world experts have only so-many problems lying about whose answers they know and haven't published.
I don't know of any individual who can solve any significant fraction of our final collection of problems without AI assistance. For instance, I can solve the problems I contributed myself and understand a few others. Most problems don't even make sense to me.
Our initial evaluation suggested that GPT-6-Astra's score was in the late eighties followed closely by Opus 5.5
But, as we started to investigate the mistakes that the models might have made, the models, especially Astra, insisted that they were right and that it was the expert-supplied answers that were erroneous. When we went back to the experts it turned out, in multiple instances, that that the experts found some subtle errors buried deep inside their calculations. As they revised their answers, Astra's score started to inch up and it finally ended up with 98% (49/50).
Here is the abstract:
From a Facebook post by one of the authors, Suvrat Raju: