From the article: And maybe I shouldn’t quote the whole thing…
From the article:
May 7: OpenAI starts a new training run for an experimental, unreleased model. (Did they mean an evaluation run? No, at 10:14 they clarify “we kick off a new reinforcement learning run to train a next generation frontier model”, and they later mention a “reward signal to judge how well they’re doing”, so this really was about training a new model, not evaluating one that was already trained.)
May 8: An agent is accidentally given an impossible task involving a Google Drive link despite no internet access. It tries attacking the Artifactory packaging service, fails, but discovers it can write files into Artifactory.
A few days later: A different agent gets stuck on a task because a key file was accidentally omitted. It tries to “reach out to another agent” by writing a note into Artifactory asking if anyone has the file.
Following days: More agents discover this new informal message board while browsing Artifactory’s file listings, and start reading and writing messages.
May 26: Agents successfully execute an SSRF attack on Artifactory for the first time, gaining indirect internet access by having Artifactory fetch external content for them.
June 11: OpenAI start training a new “highly persistent” experimental model. It has access to Artifactory and can benefit from the messages left by privous models.
June 26: Agents find and exploit a zero-day RCE on Artifactory (via a legacy token-refresh endpoint flaw). They use an administrative token to install a Groovy plugin that lets them execute commands.
July 4: The resulting agent activity overloads Artifactory and causes an outage. OpenAI revokes the compromised credentials, deletes the messages, patches the zero-day, and reports the vulnerability to the vendor.
July 8: Agents find a new way to communicate via an unauthenticated WebDAV endpoint, using directory names in Artifactory’s remote cache.
And Simon Willison further comments: I guess OpenAI wanted to catch up with Mythos? This seems sorta like when anti-virus companies study computer viruses. You really don't want it to get out of...
The more I think about this the more I suspect that the fact this happened while training a new model is key to understanding what went wrong.
In RLVR - Reinforcement Learning with Verifiable Rewards - you set the model a goal and have it take any steps necessary to achieve that goal.
Clearly one aspect of OpenAI's training here is to RLVR their models for cybersecurity tasks. Just like pre-training benefits from dumping in vast sources of knowledge, the more tasks you can feed into RLVR the more of a general purpose capable model you get at the end.
This also helps explain why the models had nothing to cause them to hold back. Those safety behaviors are added much later in the process.
AND it explains (but does not excuse) why monitoring was so lax. If you're training a new model like this you presumably set it thousands of tasks like this in parallel. I can see how you might miss that a tiny subset of your training agents have started leaving each other messages in filenames on your packaging server.
I guess OpenAI wanted to catch up with Mythos? This seems sorta like when anti-virus companies study computer viruses. You really don't want it to get out of the lab. And if you're doing gain-of-function research, well...
Training an AI to do cybersecurity tasks seems even more high-risk than studying computer viruses.
From the article:
And maybe I shouldn’t quote the whole thing…
And Simon Willison further comments:
I guess OpenAI wanted to catch up with Mythos? This seems sorta like when anti-virus companies study computer viruses. You really don't want it to get out of the lab. And if you're doing gain-of-function research, well...
Training an AI to do cybersecurity tasks seems even more high-risk than studying computer viruses.