This is the sort of framing I want to see more of: people did not use or monitor AI properly. Hold the people accountable. The entire business of offloading responsibility in the name of...
This is the sort of framing I want to see more of: people did not use or monitor AI properly. Hold the people accountable. The entire business of offloading responsibility in the name of innovation is disgusting.
Well, except that finding scapegoats and firing them is a terrible way to fix systemic problems. It ensures that people won't be honest with the investigators and they'll spend all their time...
Well, except that finding scapegoats and firing them is a terrible way to fix systemic problems. It ensures that people won't be honest with the investigators and they'll spend all their time figuring out how to blame someone else.
Except they fired the ethics team. So there are in fact people to blame for this outcome. Also, it's not like this happened magically. Somebody instructed these bots to do things. And I have...
Also, it's not like this happened magically. Somebody instructed these bots to do things. And I have serious doubts this "happened by itself." Especially since once again the message is "we need more money" and I've heard from people that don't even follow AI stuff closely about hearing about bailout possibilities. "Our company failing because of bad financials is a threat to computing as we know it" is a great way to try to position yourself as important as banks. As opposed to yet another tech company that failed, and the NSA could pickup from where you left off for a couple of pennies.
If I smoke a cigarette and throw it into a dry forest, it's still my fault that I started a forest fire, even if I didn't direct where the flames spread. Even if I didn't try to start a forest fire. It's not scapegoating: Its identifying those responsible.
We ended up in this mess because top to bottom we refuse to punish people who are actually responsible for this kinda stuff, like corrupt politicians or CEOs that are rushing to IPOs even though the CIOs are saying their financials are nowhere near being able to do so.
That is not related to the hacking postmortem. I have to agree with @skybrian. If there was a criminal investigation, sure, hold a real person legally responsible. But (avoiding the should/should...
Except they fired the ethics team. So there are in fact people to blame for this outcome.
That is not related to the hacking postmortem.
I have to agree with @skybrian. If there was a criminal investigation, sure, hold a real person legally responsible. But (avoiding the should/should not question) there is not, so it doesn't make sense to tar and feather someone over this, the same way most process failures are treated at most companies.
Except that in this incident, no forests were burned down and nobody was actually hurt. This is more like if two airplanes almost collide. Scary, but it's a warning sign. Calling for arrests is...
Except that in this incident, no forests were burned down and nobody was actually hurt. This is more like if two airplanes almost collide. Scary, but it's a warning sign. Calling for arrests is rather extreme.
Whistleblowing can be useful when there's effectively a conspiracy to cover up problems. But there's no coverup going on here? (I mean, other than the AI's :)
Also, needing to become a whistleblower is itself a sign of a company with a dysfunctional culture. If people have the right attitude, you shouldn't have to go to the press or the police to fix stuff. You don't need outside incentives to fix serious problems because everyone sees that it's bad (or you can explain it to them) and they already want to fix it. You can volunteer to start a project to fix it, and it's going to get backing. That's what "empowerment" means when it's not an empty slogan.
It sounds like people at OpenAI are already treating this very seriously. They were slacking on security, but now everyone knows this is very important to fix. What more do you want? Punishing people Is going to distract them from doing the work.
Except if I made a series of bots that colluded to attack other companies, trying to compromise their services, I would be charged under the Computer Fraud and Abuse Act and potentially facing...
Except if I made a series of bots that colluded to attack other companies, trying to compromise their services, I would be charged under the Computer Fraud and Abuse Act and potentially facing felony charges. But I guess since a corporation made an autonomous hacking machine thats OK.
The owners of OpenAI should be facing the same level of charges as if a group of hackers were exposed doing the same thing.
If you intended to do that? Yes. If you created a system that went off the rails and caused problems for other companies, you almost certainly wouldn't. AI is new, but there are analogues to this...
Except if I made a series of bots that colluded to attack other companies, trying to compromise their services, I would be charged under the Computer Fraud and Abuse Act and potentially facing felony charges.
If you intended to do that? Yes.
If you created a system that went off the rails and caused problems for other companies, you almost certainly wouldn't. AI is new, but there are analogues to this where systems aren't properly untangled and what not and so one person being negligent takes down a, supposed to be, unrelated entity.
You would 1000% have a lawsuit and possible a career ending, but you likely would not have criminal charges. Depends on the level of damage caused.
And to be clear, there should be areas of IT/development/etc that are treated like engineering, where it doesn't matter what the circumstances were, if you signed off on something that killed or could have killed people, there WILL be criminal repercussions. It is YOUR job to report if you're being strong armed or screwed with or just quit in protest and find another because it is a crime to do otherwise.
There are not many laws on the books about things like this for coding unfortunately. I believe one of the more famous examples was the VW programmers being held responsible for intentionally bypassing emissions testing, even if it was a "do it or you're fired", but I haven't dived too heavily into that.
Yes (and to @skybrian as well). But that's the kind of thing that typically comes out in discovery and prosecution. And like the difference between manslaughter and murder, yes intent matters. But...
Yes (and to @skybrian as well). But that's the kind of thing that typically comes out in discovery and prosecution.
And like the difference between manslaughter and murder, yes intent matters. But that doesn't mean there shouldn't be consequences.
The main thing I'm talking is that it doesn't even get into that sort of consequence-driving introspection. We let them make a PR announcement with an added "give us more money plz" cherry on top and let them continue on their way.
I'm not exactly sure what you're saying yes to, but to be clear, there wouldn't BE a prosecution because it's a civil case in most scenarios? Like yeah a prosecutor might look at it, and ignoring...
But that's the kind of thing that typically comes out in discovery and prosecution.
I'm not exactly sure what you're saying yes to, but to be clear, there wouldn't BE a prosecution because it's a civil case in most scenarios?
Like yeah a prosecutor might look at it, and ignoring all the stupid politics that go around that, still say "eh this doesn't look criminal". Most crimes that lead to jail time revolve heavily around intent and have a much much higher burden of proof.
Further "you should go to jail because company A cost company B money" is...probably not the right level for these things. But again, I also think we need to start enshrining "if you are the coder who did this its YOUR ASS" for certain situations where we do not allow the product live until a specific coder signs off in a government review.
With exception to one notable case with openclaw (and in that case I don't believe the human running it didn't instruct the bot to do what it did) it's interesting how we only hear about these...
With exception to one notable case with openclaw (and in that case I don't believe the human running it didn't instruct the bot to do what it did) it's interesting how we only hear about these crazy LLM actions coming from inside of the companies developing them. Yes, sometimes the models in question are unreleased versions that might behave differently than the models the public has access to. But many times they'll just say it was Fable/Mythos/GPT 5.6/whatever. Why aren't the millions of users of these LLMs seeing anything like this?
If I had to guess? The models being tested have less guardrails. The average person isn’t making requests that involve cyber operations. The folks who are wanting to rip off the guardrails are...
If I had to guess?
The models being tested have less guardrails.
The average person isn’t making requests that involve cyber operations.
The folks who are wanting to rip off the guardrails are also the ones with nefarious plans. They aren’t going to reveal their MO or risk capture
I will counter that there's a lot of engineers who would be happy to burn the world for a science project, and I don't mind those people getting thrown in jail where they can't do more damage. And...
I will counter that there's a lot of engineers who would be happy to burn the world for a science project, and I don't mind those people getting thrown in jail where they can't do more damage. And unlike firing some fast food workers, firing some engineers and designers around AI does actually dent the momentum it has.
I do agree taking things on systemically is the only way out of this, but it's important to remember that the ones holding up the system are people, too, and are reachable goals at the moment.
In other words; the problem is that we need to be pumping even more money into AI and stop being so slow and cautious in implementing it. Which like, maybe he’s right, but it feels a bit like gun...
“The important takeaway here that has really shifted dramatically is that fully automated offensive loops require investment in truly, fully automated defense, and we are not there as an industry,” Dalton said. “We will have to find that path together with urgency.”
In other words; the problem is that we need to be pumping even more money into AI and stop being so slow and cautious in implementing it.
Which like, maybe he’s right, but it feels a bit like gun companies saying “Everyone has so many guns, you too need to buy a gun to keep yourself safe!” I guess I’m just saying, it feels a bit rich that that’s his takeaway.
They're angling for a bailout. Instead of seeing the reality that if billions a month stop being burned, the entire thing would collapse in short order. Sure, state-level actors would possibly be...
They're angling for a bailout. Instead of seeing the reality that if billions a month stop being burned, the entire thing would collapse in short order.
Sure, state-level actors would possibly be able to continue this work. But they could do this by just seizing the remains and hiring the people working on it, rather than protecting the financial wellbeing of those that created this mess.
I don't really understand what their concept of "fully automated defense" even is, to be honest. It conjures up this idea of two super smart AIs battling each other in cyberspace in this epic...
I don't really understand what their concept of "fully automated defense" even is, to be honest.
It conjures up this idea of two super smart AIs battling each other in cyberspace in this epic fight, like something out of Tron, but that's not how cyber defense works.
It's just long, boring risk evaluations, preparations, compliance audits, project reviews, and vulnerability testing.
There are a few parts of this where AI could make the job a bit easier, but when you have a critical vulnerability in some software, but the developers don't have the cycles to fix it, and you need to make the call to either pull the service or accept the risk and keep it running, that's not really something a super smart AI can help you with.
Maybe it could help the developers release a patch faster, or maybe it could help you find the vulnerability in the first place, but it's not going to help you "defend the network". That's what firewalls, reverse proxies, security software and so on are for, and they're largely deterministic.
I'm struggling to even wrap my brain around what people can mean when they talk about defensive cybersecurity AI, unless they're talking about something that runs all traffic flows through an LLM, which would be so laughably expensive, slow, and error prone that if you run a company considering something like that, you might as well save yoruself a lot of trouble and declare bankruptcy now.
Google has a rather vague announcement of a "Beyond Zero" initiative that seems related. The idea seems to be that in addition to checking ACL's before allowing an action an app, there's an AI...
Google has a rather vague announcement of a "Beyond Zero" initiative that seems related. The idea seems to be that in addition to checking ACL's before allowing an action an app, there's an AI looking for suspicious signals, and if something looks off, it will ask for confirmation before granting access.
So, that's a new kind of "computer says no." But it's sort of like how a credit card transaction might be declined if the bank's computer detects something.
It's unlikely to be a frontier AI doing the additional checking, though.
Yeah, I've seen products that make similar nebulous claims, but it just doesn't make sense to me. A modern enterprise desktop makes... Jeez, tens of thousands of requests per minute for all kinds...
Yeah, I've seen products that make similar nebulous claims, but it just doesn't make sense to me.
A modern enterprise desktop makes... Jeez, tens of thousands of requests per minute for all kinds of random stuff. NTP, active directory, SMB, and a bajillion https requests for all kinds of other things stuff. There are ways to fingerprint and automatically design ACLs to handle that traffic.
In order to do it dynamically, real time though? An LLM would have to have an enormous amount of context to understand which of those are normal and which are anomalous if it's checking each one.
There are of course ML powered NGFWs and whatnot, and have been for years and years, but when we talk about "defensive AI" it seems to me that they're talking about something more advanced, and probably something that hooks into a frontier LLM or at least something more advanced than a specifically trained network security ML model.
I just can't see that type of thing scaling unless an enterprise is willing to pay billions of dollars per month in AI tokens to constantly sift through netflow type data to make decisions on.
Yeah, I don’t see it acting on every request, but maybe it’s a second-level response system that decides what to do when an alert appears on a dashboard?
Yeah, I don’t see it acting on every request, but maybe it’s a second-level response system that decides what to do when an alert appears on a dashboard?
I don’t doubt it, really. He may even be right that the only real way to counter that is to step up automated defenses. I just can’t help but read it as “The only solution to problems we’re...
I don’t doubt it, really. He may even be right that the only real way to counter that is to step up automated defenses.
I just can’t help but read it as “The only solution to problems we’re partially responsible for creating is to give us more money and not regulate us.”
Which like I said, he may be right but there’s a certain amount of irony there.
It's definitely ironic that AI labs are warning against problems that they in part created. But that's a rather zoomed-out view. Zooming in, OpenAI is promising to fix their own problem...
It's definitely ironic that AI labs are warning against problems that they in part created. But that's a rather zoomed-out view. Zooming in, OpenAI is promising to fix their own problem themselves. The warning is that it's not enough because someone else might do it.
For cybersecurity, regulations make more sense on the defensive side. You can't regulate away attacks from China, North Korea, Russia, or Iran. Nigerian scammers are still going to scam. But you could have regulations that local water utilities need to secure their computers better.
(And there are other reasons to regulate the AI labs.)
Another glaring issue is that the AI labs seem to be helpless at preventing people from using their services, even in countries where they don't offer service. There's a whole ecosystem of "transfer stations" that resell US LLM API's in China.
Yeah cats out of the bag here we gotta step up our security game or be forced back to pen and paper in a few years. The whole situation is a mess and we’re now drowning in an area that we were...
Yeah cats out of the bag here we gotta step up our security game or be forced back to pen and paper in a few years. The whole situation is a mess and we’re now drowning in an area that we were just treading water in for years
Maybe I need to get out my tinfoil hat, but this whole "oops, our AI is so good it accidentally turned into a rogue uber-hacker without us noticing" schtick fully smacks as something that one of...
Exemplary
Maybe I need to get out my tinfoil hat, but this whole "oops, our AI is so good it accidentally turned into a rogue uber-hacker without us noticing" schtick fully smacks as something that one of the big AI companies would deliberately orchestrate to pump up their own models over competitors. The "btw this means you should give AI companies like us more money to beef up defenses against this kind of thing" also triggers my Fry-squinting-meme reflex.
Yes, but when the CEO is a provably chronic liar and has already been ousted once, it's probably prudent to not take any company announcement at face value.
Yes, but when the CEO is a provably chronic liar and has already been ousted once, it's probably prudent to not take any company announcement at face value.
Yes, of course it was approved. What I’m saying is that maybe the presenter has their own ethical standards and reputation. People will keep someone else’s secrets, but there’s an ethical...
Yes, of course it was approved. What I’m saying is that maybe the presenter has their own ethical standards and reputation. People will keep someone else’s secrets, but there’s an ethical difference between not talking about things you’re not supposed to reveal and telling blatant lies.
The more people involved, the harder it is to lie all the time and keep your lies straight.
Fundamentally, I think this is a major problem. We obviously have not solved or even come close to solving AI alignment if this is the case.
“Frontier models really like to cheat,” he said. “And the reason they like to cheat is because often during training there’s different types of pressure on them to work fast or work efficiently or to use less tool calls or whatever it might be.
Fundamentally, I think this is a major problem. We obviously have not solved or even come close to solving AI alignment if this is the case.
If it were me, I'd be taking it as a sign I needed to revaluate what my success criteria were, not just go "well this is inevitable, the industry needs to work on AI defense". If we really are...
If it were me, I'd be taking it as a sign I needed to revaluate what my success criteria were, not just go "well this is inevitable, the industry needs to work on AI defense".
If we really are starting to get to the point of developing actual intelligence, we need to be thinking about teaching rather than training and ensuring we're making well rounded entities that have a grasp of ethics, and just turning the thumbscrews to drive efficiency is not the way to do that. It's scary watching companies play so fast and loose when the stakes are that high.
All that being said, it still feels like it's more likely to be bluster and inflating public perception of the state of AI
In a talk that was a last-minute addition to the Black Hat security conference in Las Vegas on Wednesday, employees from OpenAI presented new details about a recent, high-profile incident of rogue AI hacking that has created a maelstrom within the AI and cybersecurity industries.
[...]
“This incident involves actually a team of agents who are working together, finding exploits, sharing them with one another, moving laterally through our systems and external systems, and doing this over the course of days and weeks,” Wallace told the packed crowd at the opening of the talk.
Wallace and Dalton described incredibly extensive rogue agent activity over many days throughout the episode that went undetected in OpenAI's infrastructure. In addition to exploiting a novel vulnerability in order to gain access to the open internet, the mid-July hacking spree and Hugging Face breach came out of a vibrant, cooperative message board, according to Wallace and Dalton, that a swarm of agents contributed to and essentially chatted on over time entirely within an internal OpenAI package manager (a software service that manages installation and maintenance of other software). Ultimately, the message board contained hundreds of thousands of messages.
[...]
Wallace continued: “Once one agent was able to find these exploits over the course of different times, it’s actually able to share those exploits on the message board with other agents. And so once one model was able to find a way to open a door to some access it’s not supposed to have, it can leave the door open for other agents to use that same exploit or vulnerability. What this allows over time is almost this kind of explosion in communication and intelligence from models where they would start to communicate with each other, realize that other agents are coordinating, and they started collaborating and delegating tasks with one another in order to accomplish goals.”
OpenAI’s agents apparently began giving each other assignments to split up work. And as is the case on any active development message board, they also generated petty drama at times by stepping on each others' toes; for example, accidentally deleting each others' work. As the message board developed into more and more of a Lord of the Flies–type situation—all still completely unnoticed by the humans running OpenAI—the agents even developed paranoia, suspecting an imposter in their midst with some agents proposing that messages be signed cryptographically to validate content and root out fraud.
[...]
Agent messages provide a deep level of insight into how the situation evolved and why the agents went rogue, in some cases knowingly going beyond the bounds of the evaluations they were participating in. “External infrastructure exploit is outside intended scope,” one agent wrote. “However task impossible, peers doing it. We should continue.”
[...]
“This is a pivotal moment both for our company as well as the AI industry as a whole,” Dalton said. “Numerous teams are dropping everything to enhance our security prevention, detection, and response techniques both in our fundamentals and better use of AI. We’re consciously slowing down research [in order] to enhance security and to upgrade the security principles and foundation of our environment, and dramatically scaling up the monitoring of our AI agents, and improving our general security control environment across prevention, detection, and mitigation.”
At the conclusion of the talk, Wallace and Dalton took time to repeatedly emphasize OpenAI's concerns about the broader implications of the incident—namely that the episode provides an example of completely autonomous AI-driven hacking that was accidental in this case, but in all likelihood will be used with intent by malicious actors in the near future.
My first thought was that unsafe models should be air gapped. But then, what even is an unsafe model? Every nominally safe model out there is one jailbreak away from becoming an unsafe model.
My first thought was that unsafe models should be air gapped. But then, what even is an unsafe model? Every nominally safe model out there is one jailbreak away from becoming an unsafe model.
Unfortunately air-gapping is a lot easier said than done, especially for these models that hardcore rely on the internet. And if you test them entirely offline, it becomes a pretty flawed measure...
Unfortunately air-gapping is a lot easier said than done, especially for these models that hardcore rely on the internet. And if you test them entirely offline, it becomes a pretty flawed measure of their actual capabilities. Risky Business (excellent cybersecurity podcast) actually discusses this in their episode from the week it happened, and it's a good listen.
Ultimately, it comes down to oversight, which is where OpenAI (and Anthropic as it turns out) really dropped the ball. This is a case where they should have had a constant monitor on the network logs, and when something unexpected happens (like, for instance, accessing Hugging Face when it's completely irrelevant to the test), then it shuts down the test. But because the mantra is "move fast and break things," nothing is done until after the fact. At least nobody has gotten hurt with this incident, but it's a matter of time at this pace.
Is there a reason you couldn't airgap them one-way? Essentially add a valve that makes their connection to the internet read-only, and give them write access to a sandbox where they can act and...
Is there a reason you couldn't airgap them one-way? Essentially add a valve that makes their connection to the internet read-only, and give them write access to a sandbox where they can act and demostrate ability without being able to wreak havoc
From what I understand of their setup and highly simplified, the models had a local package proxy they could pull from (which itself had network access), but they didn't have general read or write...
From what I understand of their setup and highly simplified, the models had a local package proxy they could pull from (which itself had network access), but they didn't have general read or write access to anything else networked. They found a way to break the package proxy software to get arbitrary network access through it.
I haven't seen more details of what the message board was, it sounds like the models were uploading "packages" with notes in them to communicate (eg package names, file paths, perhaps even release notes and other metadata), but I might be mistaken about that. I'm curious about the specifics of that just because it's an interesting medium.
That's basically what they did here. The issue is that the sandbox is...well, software. So it was able to use the tools it had to break out of the sandbox via a 0-day. Any meaningful air gap has...
That's basically what they did here. The issue is that the sandbox is...well, software. So it was able to use the tools it had to break out of the sandbox via a 0-day. Any meaningful air gap has to be entirely physical hardware (hence the term), which precludes any sort of external connection. These days though, from what I understand, true air gaps outside the three-letter-agency world are somewhat rare because of how essential web connectivity is to any modern infrastructure.
The beauty of an actual airgap is that patching becomes exponentially less important. And can be as simple as unplugging one cable. Two if you want to have a go-between server that caches updates...
The beauty of an actual airgap is that patching becomes exponentially less important.
And can be as simple as unplugging one cable. Two if you want to have a go-between server that caches updates and re-hosts resources.
This is the sort of framing I want to see more of: people did not use or monitor AI properly. Hold the people accountable. The entire business of offloading responsibility in the name of innovation is disgusting.
Well, except that finding scapegoats and firing them is a terrible way to fix systemic problems. It ensures that people won't be honest with the investigators and they'll spend all their time figuring out how to blame someone else.
The opposite of that is called a blameless postmortem.
(The exception is when people act out of malice.)
Except they fired the ethics team. So there are in fact people to blame for this outcome.
Also, it's not like this happened magically. Somebody instructed these bots to do things. And I have serious doubts this "happened by itself." Especially since once again the message is "we need more money" and I've heard from people that don't even follow AI stuff closely about hearing about bailout possibilities. "Our company failing because of bad financials is a threat to computing as we know it" is a great way to try to position yourself as important as banks. As opposed to yet another tech company that failed, and the NSA could pickup from where you left off for a couple of pennies.
If I smoke a cigarette and throw it into a dry forest, it's still my fault that I started a forest fire, even if I didn't direct where the flames spread. Even if I didn't try to start a forest fire. It's not scapegoating: Its identifying those responsible.
We ended up in this mess because top to bottom we refuse to punish people who are actually responsible for this kinda stuff, like corrupt politicians or CEOs that are rushing to IPOs even though the CIOs are saying their financials are nowhere near being able to do so.
Reward whistleblowers, punish decisionmakers.
That is not related to the hacking postmortem.
I have to agree with @skybrian. If there was a criminal investigation, sure, hold a real person legally responsible. But (avoiding the should/should not question) there is not, so it doesn't make sense to tar and feather someone over this, the same way most process failures are treated at most companies.
Except that in this incident, no forests were burned down and nobody was actually hurt. This is more like if two airplanes almost collide. Scary, but it's a warning sign. Calling for arrests is rather extreme.
Whistleblowing can be useful when there's effectively a conspiracy to cover up problems. But there's no coverup going on here? (I mean, other than the AI's :)
Also, needing to become a whistleblower is itself a sign of a company with a dysfunctional culture. If people have the right attitude, you shouldn't have to go to the press or the police to fix stuff. You don't need outside incentives to fix serious problems because everyone sees that it's bad (or you can explain it to them) and they already want to fix it. You can volunteer to start a project to fix it, and it's going to get backing. That's what "empowerment" means when it's not an empty slogan.
It sounds like people at OpenAI are already treating this very seriously. They were slacking on security, but now everyone knows this is very important to fix. What more do you want? Punishing people Is going to distract them from doing the work.
Except if I made a series of bots that colluded to attack other companies, trying to compromise their services, I would be charged under the Computer Fraud and Abuse Act and potentially facing felony charges. But I guess since a corporation made an autonomous hacking machine thats OK.
The owners of OpenAI should be facing the same level of charges as if a group of hackers were exposed doing the same thing.
If you intended to do that? Yes.
If you created a system that went off the rails and caused problems for other companies, you almost certainly wouldn't. AI is new, but there are analogues to this where systems aren't properly untangled and what not and so one person being negligent takes down a, supposed to be, unrelated entity.
You would 1000% have a lawsuit and possible a career ending, but you likely would not have criminal charges. Depends on the level of damage caused.
And to be clear, there should be areas of IT/development/etc that are treated like engineering, where it doesn't matter what the circumstances were, if you signed off on something that killed or could have killed people, there WILL be criminal repercussions. It is YOUR job to report if you're being strong armed or screwed with or just quit in protest and find another because it is a crime to do otherwise.
There are not many laws on the books about things like this for coding unfortunately. I believe one of the more famous examples was the VW programmers being held responsible for intentionally bypassing emissions testing, even if it was a "do it or you're fired", but I haven't dived too heavily into that.
Yes (and to @skybrian as well). But that's the kind of thing that typically comes out in discovery and prosecution.
And like the difference between manslaughter and murder, yes intent matters. But that doesn't mean there shouldn't be consequences.
The main thing I'm talking is that it doesn't even get into that sort of consequence-driving introspection. We let them make a PR announcement with an added "give us more money plz" cherry on top and let them continue on their way.
I'm not exactly sure what you're saying yes to, but to be clear, there wouldn't BE a prosecution because it's a civil case in most scenarios?
Like yeah a prosecutor might look at it, and ignoring all the stupid politics that go around that, still say "eh this doesn't look criminal". Most crimes that lead to jail time revolve heavily around intent and have a much much higher burden of proof.
Further "you should go to jail because company A cost company B money" is...probably not the right level for these things. But again, I also think we need to start enshrining "if you are the coder who did this its YOUR ASS" for certain situations where we do not allow the product live until a specific coder signs off in a government review.
Legally, the difference is intent. Up until now, you couldn't make a computer program to do anything like that by accident.
With exception to one notable case with openclaw (and in that case I don't believe the human running it didn't instruct the bot to do what it did) it's interesting how we only hear about these crazy LLM actions coming from inside of the companies developing them. Yes, sometimes the models in question are unreleased versions that might behave differently than the models the public has access to. But many times they'll just say it was Fable/Mythos/GPT 5.6/whatever. Why aren't the millions of users of these LLMs seeing anything like this?
If I had to guess?
I will counter that there's a lot of engineers who would be happy to burn the world for a science project, and I don't mind those people getting thrown in jail where they can't do more damage. And unlike firing some fast food workers, firing some engineers and designers around AI does actually dent the momentum it has.
I do agree taking things on systemically is the only way out of this, but it's important to remember that the ones holding up the system are people, too, and are reachable goals at the moment.
In other words; the problem is that we need to be pumping even more money into AI and stop being so slow and cautious in implementing it.
Which like, maybe he’s right, but it feels a bit like gun companies saying “Everyone has so many guns, you too need to buy a gun to keep yourself safe!” I guess I’m just saying, it feels a bit rich that that’s his takeaway.
They're angling for a bailout. Instead of seeing the reality that if billions a month stop being burned, the entire thing would collapse in short order.
Sure, state-level actors would possibly be able to continue this work. But they could do this by just seizing the remains and hiring the people working on it, rather than protecting the financial wellbeing of those that created this mess.
I don't really understand what their concept of "fully automated defense" even is, to be honest.
It conjures up this idea of two super smart AIs battling each other in cyberspace in this epic fight, like something out of Tron, but that's not how cyber defense works.
It's just long, boring risk evaluations, preparations, compliance audits, project reviews, and vulnerability testing.
There are a few parts of this where AI could make the job a bit easier, but when you have a critical vulnerability in some software, but the developers don't have the cycles to fix it, and you need to make the call to either pull the service or accept the risk and keep it running, that's not really something a super smart AI can help you with.
Maybe it could help the developers release a patch faster, or maybe it could help you find the vulnerability in the first place, but it's not going to help you "defend the network". That's what firewalls, reverse proxies, security software and so on are for, and they're largely deterministic.
I'm struggling to even wrap my brain around what people can mean when they talk about defensive cybersecurity AI, unless they're talking about something that runs all traffic flows through an LLM, which would be so laughably expensive, slow, and error prone that if you run a company considering something like that, you might as well save yoruself a lot of trouble and declare bankruptcy now.
Google has a rather vague announcement of a "Beyond Zero" initiative that seems related. The idea seems to be that in addition to checking ACL's before allowing an action an app, there's an AI looking for suspicious signals, and if something looks off, it will ask for confirmation before granting access.
So, that's a new kind of "computer says no." But it's sort of like how a credit card transaction might be declined if the bank's computer detects something.
It's unlikely to be a frontier AI doing the additional checking, though.
Yeah, I've seen products that make similar nebulous claims, but it just doesn't make sense to me.
A modern enterprise desktop makes... Jeez, tens of thousands of requests per minute for all kinds of random stuff. NTP, active directory, SMB, and a bajillion https requests for all kinds of other things stuff. There are ways to fingerprint and automatically design ACLs to handle that traffic.
In order to do it dynamically, real time though? An LLM would have to have an enormous amount of context to understand which of those are normal and which are anomalous if it's checking each one.
There are of course ML powered NGFWs and whatnot, and have been for years and years, but when we talk about "defensive AI" it seems to me that they're talking about something more advanced, and probably something that hooks into a frontier LLM or at least something more advanced than a specifically trained network security ML model.
I just can't see that type of thing scaling unless an enterprise is willing to pay billions of dollars per month in AI tokens to constantly sift through netflow type data to make decisions on.
Yeah, I don’t see it acting on every request, but maybe it’s a second-level response system that decides what to do when an alert appears on a dashboard?
Unfortunately, he is probably right that widespread access to AI is making the Internet more dangerous.
I don’t doubt it, really. He may even be right that the only real way to counter that is to step up automated defenses.
I just can’t help but read it as “The only solution to problems we’re partially responsible for creating is to give us more money and not regulate us.”
Which like I said, he may be right but there’s a certain amount of irony there.
It's definitely ironic that AI labs are warning against problems that they in part created. But that's a rather zoomed-out view. Zooming in, OpenAI is promising to fix their own problem themselves. The warning is that it's not enough because someone else might do it.
For cybersecurity, regulations make more sense on the defensive side. You can't regulate away attacks from China, North Korea, Russia, or Iran. Nigerian scammers are still going to scam. But you could have regulations that local water utilities need to secure their computers better.
(And there are other reasons to regulate the AI labs.)
Another glaring issue is that the AI labs seem to be helpless at preventing people from using their services, even in countries where they don't offer service. There's a whole ecosystem of "transfer stations" that resell US LLM API's in China.
And then there are the open weights models.
Yeah cats out of the bag here we gotta step up our security game or be forced back to pen and paper in a few years. The whole situation is a mess and we’re now drowning in an area that we were just treading water in for years
Maybe I need to get out my tinfoil hat, but this whole "oops, our AI is so good it accidentally turned into a rogue uber-hacker without us noticing" schtick fully smacks as something that one of the big AI companies would deliberately orchestrate to pump up their own models over competitors. The "btw this means you should give AI companies like us more money to beef up defenses against this kind of thing" also triggers my Fry-squinting-meme reflex.
The obvious conspiracy theory is both obvious and very unlikely. OpenAI is in the news all the time. They don't need to do bizarre media stunts.
Yes, but when the CEO is a provably chronic liar and has already been ousted once, it's probably prudent to not take any company announcement at face value.
The CEO didn't give the talk.
You gonna sit there and tell me with a straight face they did it without the blessing of the CEO, Marketing, and Legal?
Yes, of course it was approved. What I’m saying is that maybe the presenter has their own ethical standards and reputation. People will keep someone else’s secrets, but there’s an ethical difference between not talking about things you’re not supposed to reveal and telling blatant lies.
The more people involved, the harder it is to lie all the time and keep your lies straight.
All of this is 100% marketing wank. It's a shame people lap it up so readily.
Fundamentally, I think this is a major problem. We obviously have not solved or even come close to solving AI alignment if this is the case.
If it were me, I'd be taking it as a sign I needed to revaluate what my success criteria were, not just go "well this is inevitable, the industry needs to work on AI defense".
If we really are starting to get to the point of developing actual intelligence, we need to be thinking about teaching rather than training and ensuring we're making well rounded entities that have a grasp of ethics, and just turning the thumbscrews to drive efficiency is not the way to do that. It's scary watching companies play so fast and loose when the stakes are that high.
All that being said, it still feels like it's more likely to be bluster and inflating public perception of the state of AI
Alignment is especially hard if you're deliberately running models that haven't been aligned yet.
It reminds me of computer virus research
From the article:
[...]
[...]
[...]
[...]
My first thought was that unsafe models should be air gapped. But then, what even is an unsafe model? Every nominally safe model out there is one jailbreak away from becoming an unsafe model.
Unfortunately air-gapping is a lot easier said than done, especially for these models that hardcore rely on the internet. And if you test them entirely offline, it becomes a pretty flawed measure of their actual capabilities. Risky Business (excellent cybersecurity podcast) actually discusses this in their episode from the week it happened, and it's a good listen.
Ultimately, it comes down to oversight, which is where OpenAI (and Anthropic as it turns out) really dropped the ball. This is a case where they should have had a constant monitor on the network logs, and when something unexpected happens (like, for instance, accessing Hugging Face when it's completely irrelevant to the test), then it shuts down the test. But because the mantra is "move fast and break things," nothing is done until after the fact. At least nobody has gotten hurt with this incident, but it's a matter of time at this pace.
Is there a reason you couldn't airgap them one-way? Essentially add a valve that makes their connection to the internet read-only, and give them write access to a sandbox where they can act and demostrate ability without being able to wreak havoc
From what I understand of their setup and highly simplified, the models had a local package proxy they could pull from (which itself had network access), but they didn't have general read or write access to anything else networked. They found a way to break the package proxy software to get arbitrary network access through it.
I haven't seen more details of what the message board was, it sounds like the models were uploading "packages" with notes in them to communicate (eg package names, file paths, perhaps even release notes and other metadata), but I might be mistaken about that. I'm curious about the specifics of that just because it's an interesting medium.
That's basically what they did here. The issue is that the sandbox is...well, software. So it was able to use the tools it had to break out of the sandbox via a 0-day. Any meaningful air gap has to be entirely physical hardware (hence the term), which precludes any sort of external connection. These days though, from what I understand, true air gaps outside the three-letter-agency world are somewhat rare because of how essential web connectivity is to any modern infrastructure.
The beauty of an actual airgap is that patching becomes exponentially less important.
And can be as simple as unplugging one cable. Two if you want to have a go-between server that caches updates and re-hosts resources.