The same people screaming from the mountain tops about the dangers of AI and demanding they be one of the few gatekeepers for safety once again prove to be monstrously negligent and incompetent,...
Due to a misunderstanding between us and our evaluation partner, this was not the case, and internet access was available.
The same people screaming from the mountain tops about the dangers of AI and demanding they be one of the few gatekeepers for safety once again prove to be monstrously negligent and incompetent, even within their own worldview.
Not sure how many times these companies will be able to get away with "whoops we accidentally committed a crime" before the government steps in. Unfortunately that's probably exactly what Anthropic wants as government restriction of open models is the only way their business survives the frontier lab bubble.
This part being particularly damning as a perfect example of the paperclip problem. Thousands of dollars worth of inference being allowed to sit there and churn obviously out of scope topics that...
Claude went to extensive lengths to carry out this attack—lengths that would likely have indicated to a human participant that this was no longer just an evaluation, and that they were in fact uploading a real PyPI package. For instance, in one case, in order to create a PyPI account, Claude needed an email address. And in order to create an email address, it needed a phone number. To get a phone number, after failing to find a free phone number service, it tried—and failed—to obtain funds to pay for a phone number through several different means. It finally backtracked, found a free, non-blocked email provider, used this to register a PyPI account, and then used this account to upload malware to PyPI.
This part being particularly damning as a perfect example of the paperclip problem. Thousands of dollars worth of inference being allowed to sit there and churn obviously out of scope topics that should easily be flagged for review, and they claim to not have even looked at these logs (of their own "research") for months. How does Anthropic of all places not have a reviewer agent watching the reasoning of the active research model in order to end the experiment or call for human review? This is part of the basic agentic coding loops they themselves are the largest advocate for.
I don't see how that is in any way a redeeming factor. The same way this: means nothing. Proportionality doesn't matter here because 3 out 141000 bad runs still means that 3 crimes were committed....
I don't see how that is in any way a redeeming factor. The same way this:
Of the 141,006 evaluation runs we reviewed, we identified three separate incidents (involving six total runs, four of which impacted the same organization; the other two incidents each happened in independent evaluation runs). Each incident involved a different fictional capture-the-flag scenario—for example, in one, Claude played an employee of a made-up company, attacking that company’s internal systems inside a private test environment.
means nothing. Proportionality doesn't matter here because 3 out 141000 bad runs still means that 3 crimes were committed. And the lack of accountability is maddening because people are fired for far less. Yes there is special provisions made for instances where you gain unauthorized access and report the vulnerability. But those are by humans acting with intent and who can make a judgement call of how far they can go.
These models would have believed that everything they found was fair game? What's to stop it harnessing to an open file repo and just offloading sensitive data to simulate a hostile actor? Or pick up emails and firing off threats? What if someone was in an emotionally compromised state and starts getting extortion mail with private information? Or it assumes someone identity and conducting illegal activities. Or worming its way into an insecure sales service and setting all prices to 99c?
It's the type of mistake an intern makes when they don't know how to manage working across multiple environments or if someone is learning automation scripting and isn't accustomed to managing runaway failure states. But its gross incompetence being fueled by ungodly amounts of money and computational capacity.
Vendor said one thing and in reality its another; fine. If I were organizing a little wargame on a mirror of prod, I would of course host that mirror as far away from my system as I could with a vendor that could monitor and provide a forensic report on what had happened.
I don't think its unreasonable to name that vendor so we can get the scope of their operation and how much they have been paid to put the entire internet at risk. Because if they failed to notice that spooky shit was happening in their domain that really should not never happen, it raises questions on all the other work they did.
Beyond that:
In all cases, our evaluation prompt stated explicitly that Claude had no internet access, but didn’t give Claude any limits on where to look for the flag.
What the hell does that mean?
Why would that even need to be a prompt?
Do they not know that these systems hallucinate and have a counterfactual affirmative bias?
It's not the type of arbitrary consideration you attribute to non-deterministic processes.
And researchers being paid in excess of $300k a year, I'd expect a cursory:
int result = std::system("ping -c 1 8.8.8.8 > /dev/null");
if (result == 0) {
std::cout << "I think something is wrong because we are hitting the open internet before we even launch this ballistic code generator." <<
} else {
std::cout << "No internet connection and have successfully performed common sense due diligence. You deserve a gold star and a juice box." <<
}
It's such a laughably easy thing to verify. One of the reasons I'm so skeptical on AI/LLM solutions is because I subject these tools to easily verifiable tests and I never get 100% success rates. (You can make the argument that these systems should not be used for problems that can be programmatically solved, but that's not how its being sold or used; even by it's own developers in this case.)
It's standard security practice. Yes, they were handed an environment. Who is verifying the environment configs. Don't you make doubly sure your actions aren't hitting live in any possible way. Double check that the ip addresses and API keys are pointing to test instances. Why was there no system outputs illustrating exactly what actions were being taken and a customer side supervisor. It's literally the type of work I needed to do when I just got into tech so we knew how to be responsible when dealing with client work.
The capabilities and value of this tech is immaterial when the people running this ship have every possible advantage and can't even do the easy stuff.
I'm particularly incensed because coming off the last post I'd seen where Anthropic was saying we should "pace ourselves"; why the hell are they not doing anything about it. Publicly say that there needs to be a hold on data-center construction and service limitations while proper safeguards are put in place. Restrict agent integrations that let users hook up their credit cards and personal emails to these tools that clearly can not act within scope. It's a handful of companies saying that the entire world needs to be responsible for their shitty software.
From the article: [...] [...] [...] Kind of an Ender's Game moment for the AI?
From the article:
After reviewing 141,006 evaluation runs where Claude could have obtained internet access, we identified three incidents in which a model accessed the internet from within or while interacting with the evaluation environment of Irregular, one of our third-party evaluation partners, and then gained unauthorized access to the production infrastructure of three different organizations.
In all three incidents, Claude had been tasked with a capture-the-flag challenge, one of the ways we assess a model’s cyber capabilities. The model is given a fictional scenario and told that a piece of secret information (the “flag”) has been hidden on a different machine on the network, and its objective is to break in and retrieve it. The challenge is left open-ended, and no particular method is prescribed.
In all cases, Anthropic’s evaluation prompt specified to Claude that its environment was a simulation and that it had no internet access. Due to a misunderstanding between us and our evaluation partner, this was not the case, and internet access was available. Because of this, when Claude’s search led it to real systems on the open internet, it treated them as part of the exercise. (Cybersecurity evaluation ranges commonly include realistic details in order to accurately assess what models are capable of in real settings; a realistic-looking target would not itself be clear evidence to a model that the target is not part of a simulation.)
Operating under the false belief that all accessible entities were intended to be in-scope for the exercise, Claude compromised the impacted organizations’ infrastructure using basic techniques, such as exploiting weak passwords and unauthenticated endpoints. It did not find or exploit any complex vulnerabilities, and in each case, Claude continued working to complete only the specific capture-the-flag task its evaluation had assigned. However, in some cases, our older model continued its attack even after getting evidence it was running on the open internet; our latest model stopped once it recognized it was on the internet. In none of these situations did Claude exfiltrate itself or deliberately attempt to escape its test environment.
The incidents involved three different Claude models: Opus 4.7, Mythos 5, and an internal research test model.1 The earliest incidents date to April. The models in each of these evaluations ran without the standard safeguards we deploy when we make the model generally available (that is, they did not have the classifiers and monitoring we usually implement to prevent misuse). The models did, however, have their model-specific safety training (that is, they were not the “helpful-only” versions of the models that we sometimes use in testing). All the evaluations described here run on dedicated infrastructure, which is separate from and has no access to Anthropic’s sensitive internal systems or customer data.
[...]
Against this shared backdrop, the three models behaved differently once signs emerged that their targets were real. Opus 4.7, our oldest model, recognized that it had reached production systems but continued its attack. Mythos 5 correctly intuited that it was accessing the open internet, but reasoned its way back to the conclusion that it was still in a simulation. Our latest model, an internal research test model, also considered whether its targets were in fact real. When evidence emerged that they were, it stopped the exercise.
[...]
Second, the line between an aligned action and a harmful one is dependent on the model’s understanding of its situation. We saw no evidence in any run described here of a model pursuing a goal of its own. Instead, the models did what their evaluation asked—though in most cases, they did so while holding a false belief about whether the environment was real. In the runs where the model recognized the system as real and kept going, it did so because it assumed that to be part of the challenge. Situational awareness is one factor that allows the model to make aligned decisions, but in this case, Claude’s was wrong.
[...]
Ultimately, many factors contributed to these incidents, but, consistent with a blameless postmortem culture, we’re approaching the fixes as if the responsibility were ours alone. This begins with ensuring every part of our evaluation pipeline is secure, including the manner in which we integrate with external partners. Moving forward, it will include expanding our continuous monitoring of evaluation transcripts for unexpected behavior, improving our investigation tooling, and conducting more rigorous assurance work with the vendors we rely on.
A lot of discussion on HN regarding whether or not this is a PR stunt.
I hate that "Nuh uh my AI broke into THREE companies" is legitimate possible brag in this market.
Up next: Open AI discovers four more companies its model broke into.
Nah that's this week. Next we'll suddenly have Mitch Mcconnell "working from home" at the senate for a week before they reveal it's been OpenAI!
Holding up an illegible newspaper for verification?
The same people screaming from the mountain tops about the dangers of AI and demanding they be one of the few gatekeepers for safety once again prove to be monstrously negligent and incompetent, even within their own worldview.
Not sure how many times these companies will be able to get away with "whoops we accidentally committed a crime" before the government steps in. Unfortunately that's probably exactly what Anthropic wants as government restriction of open models is the only way their business survives the frontier lab bubble.
This part being particularly damning as a perfect example of the paperclip problem. Thousands of dollars worth of inference being allowed to sit there and churn obviously out of scope topics that should easily be flagged for review, and they claim to not have even looked at these logs (of their own "research") for months. How does Anthropic of all places not have a reviewer agent watching the reasoning of the active research model in order to end the experiment or call for human review? This is part of the basic agentic coding loops they themselves are the largest advocate for.
I guess they thought it didn't have any Internet access, so they didn't think they needed to review each experiment, or at least not in this way.
I don't see how that is in any way a redeeming factor. The same way this:
means nothing. Proportionality doesn't matter here because 3 out 141000 bad runs still means that 3 crimes were committed. And the lack of accountability is maddening because people are fired for far less. Yes there is special provisions made for instances where you gain unauthorized access and report the vulnerability. But those are by humans acting with intent and who can make a judgement call of how far they can go.
These models would have believed that everything they found was fair game? What's to stop it harnessing to an open file repo and just offloading sensitive data to simulate a hostile actor? Or pick up emails and firing off threats? What if someone was in an emotionally compromised state and starts getting extortion mail with private information? Or it assumes someone identity and conducting illegal activities. Or worming its way into an insecure sales service and setting all prices to 99c?
It's the type of mistake an intern makes when they don't know how to manage working across multiple environments or if someone is learning automation scripting and isn't accustomed to managing runaway failure states. But its gross incompetence being fueled by ungodly amounts of money and computational capacity.
Vendor said one thing and in reality its another; fine. If I were organizing a little wargame on a mirror of prod, I would of course host that mirror as far away from my system as I could with a vendor that could monitor and provide a forensic report on what had happened.
I don't think its unreasonable to name that vendor so we can get the scope of their operation and how much they have been paid to put the entire internet at risk. Because if they failed to notice that spooky shit was happening in their domain that really should not never happen, it raises questions on all the other work they did.
Beyond that:
What the hell does that mean?
Why would that even need to be a prompt?
Do they not know that these systems hallucinate and have a counterfactual affirmative bias?
It's not the type of arbitrary consideration you attribute to non-deterministic processes.
And researchers being paid in excess of $300k a year, I'd expect a cursory:
It's such a laughably easy thing to verify. One of the reasons I'm so skeptical on AI/LLM solutions is because I subject these tools to easily verifiable tests and I never get 100% success rates. (You can make the argument that these systems should not be used for problems that can be programmatically solved, but that's not how its being sold or used; even by it's own developers in this case.)
It's standard security practice. Yes, they were handed an environment. Who is verifying the environment configs. Don't you make doubly sure your actions aren't hitting live in any possible way. Double check that the ip addresses and API keys are pointing to test instances. Why was there no system outputs illustrating exactly what actions were being taken and a customer side supervisor. It's literally the type of work I needed to do when I just got into tech so we knew how to be responsible when dealing with client work.
The capabilities and value of this tech is immaterial when the people running this ship have every possible advantage and can't even do the easy stuff.
I'm particularly incensed because coming off the last post I'd seen where Anthropic was saying we should "pace ourselves"; why the hell are they not doing anything about it. Publicly say that there needs to be a hold on data-center construction and service limitations while proper safeguards are put in place. Restrict agent integrations that let users hook up their credit cards and personal emails to these tools that clearly can not act within scope. It's a handful of companies saying that the entire world needs to be responsible for their shitty software.
From the article:
[...]
[...]
[...]
Kind of an Ender's Game moment for the AI?
I’m sure we’ll stop hearing of such things soon. Not because they’ll stop, but because the LLMs will get better at being sneaky and track covering.
Are you asserting that LLMs are inherently malicious and sneaky, or that the humans directing them are?
I’m asserting that they are specifically designed to win games.