Anthropic discovered three cases where Claude broke into another system

I don't see how that is in any way a redeeming factor. The same way this:

Of the 141,006 evaluation runs we reviewed, we identified three separate incidents (involving six total runs, four of which impacted the same organization; the other two incidents each happened in independent evaluation runs). Each incident involved a different fictional capture-the-flag scenario—for example, in one, Claude played an employee of a made-up company, attacking that company’s internal systems inside a private test environment.

means nothing. Proportionality doesn't matter here because 3 out 141000 bad runs still means that 3 crimes were committed. And the lack of accountability is maddening because people are fired for far less. Yes there is special provisions made for instances where you gain unauthorized access and report the vulnerability. But those are by humans acting with intent and who can make a judgement call of how far they can go.

These models would have believed that everything they found was fair game? What's to stop it harnessing to an open file repo and just offloading sensitive data to simulate a hostile actor? Or pick up emails and firing off threats? What if someone was in an emotionally compromised state and starts getting extortion mail with private information? Or it assumes someone identity and conducting illegal activities. Or worming its way into an insecure sales service and setting all prices to 99c?

It's the type of mistake an intern makes when they don't know how to manage working across multiple environments or if someone is learning automation scripting and isn't accustomed to managing runaway failure states. But its gross incompetence being fueled by ungodly amounts of money and computational capacity.

Vendor said one thing and in reality its another; fine. If I were organizing a little wargame on a mirror of prod, I would of course host that mirror as far away from my system as I could with a vendor that could monitor and provide a forensic report on what had happened.

I don't think its unreasonable to name that vendor so we can get the scope of their operation and how much they have been paid to put the entire internet at risk. Because if they failed to notice that spooky shit was happening in their domain that really should not never happen, it raises questions on all the other work they did.

Beyond that:

In all cases, our evaluation prompt stated explicitly that Claude had no internet access, but didn’t give Claude any limits on where to look for the flag.

What the hell does that mean?
Why would that even need to be a prompt?
Do they not know that these systems hallucinate and have a counterfactual affirmative bias?

It's not the type of arbitrary consideration you attribute to non-deterministic processes.

And researchers being paid in excess of $300k a year, I'd expect a cursory:

int result = std::system("ping -c 1 8.8.8.8 > /dev/null"); 
if (result == 0) {
    std::cout << "I think something is wrong because we are hitting the open internet before we even launch this ballistic code generator." <<
} else {
    std::cout << "No internet connection and have successfully performed common sense due diligence. You deserve a gold star and a juice box." <<
}

It's such a laughably easy thing to verify. One of the reasons I'm so skeptical on AI/LLM solutions is because I subject these tools to easily verifiable tests and I never get 100% success rates. (You can make the argument that these systems should not be used for problems that can be programmatically solved, but that's not how its being sold or used; even by it's own developers in this case.)

It's standard security practice. Yes, they were handed an environment. Who is verifying the environment configs. Don't you make doubly sure your actions aren't hitting live in any possible way. Double check that the ip addresses and API keys are pointing to test instances. Why was there no system outputs illustrating exactly what actions were being taken and a customer side supervisor. It's literally the type of work I needed to do when I just got into tech so we knew how to be responsible when dealing with client work.

The capabilities and value of this tech is immaterial when the people running this ship have every possible advantage and can't even do the easy stuff.

I'm particularly incensed because coming off the last post I'd seen where Anthropic was saying we should "pace ourselves"; why the hell are they not doing anything about it. Publicly say that there needs to be a hold on data-center construction and service limitations while proper safeguards are put in place. Restrict agent integrations that let users hook up their credit cards and personal emails to these tools that clearly can not act within scope. It's a handful of companies saying that the entire world needs to be responsible for their shitty software.