Seems either extremely negligent or intentionally malicious to allow or instruct semi-autonomous computer programs to hack other entities without their explicit permission. I wonder if there'll be...
Seems either extremely negligent or intentionally malicious to allow or instruct semi-autonomous computer programs to hack other entities without their explicit permission. I wonder if there'll be any consequences for that.
All reports so far indicate that it wasn't intentionally malicious and they didn't instruct their agents to do any such thing. I don't know why you threw that in there other than to muddy the waters.
All reports so far indicate that it wasn't intentionally malicious and they didn't instruct their agents to do any such thing. I don't know why you threw that in there other than to muddy the waters.
I believe I covered that with the two "or"s in my comment. As in: Obviously I have no insights into whether it was intentional or not, so either it's negligent to allow it or malicious to instruct...
I believe I covered that with the two "or"s in my comment. As in:
extremely negligent to allow
Obviously I have no insights into whether it was intentional or not, so either it's negligent to allow it or malicious to instruct it. Probably could've worded it better initially tho.
I don't know about this specific hack to Rubygems, but I watched their presentation about their hack to huggingface. I'm inclined to believe the hack to huggingface was an accident based on what...
I don't know about this specific hack to Rubygems, but I watched their presentation about their hack to huggingface. I'm inclined to believe the hack to huggingface was an accident based on what they said, the story is too "absurd" to be fake - for lack of a better word.
(Obviously, it goes without saying, it's possible they weren't 100% transparent and hid some facts.)
The TL;DR version is openAI was testing agents in an isolated environment, where they had to solve different problems. Eventually some of them reached the conclusion their problems were impossible to solve, and the answer would be on the internet. But the only thing that could reach the internet was the internal server that handed out software packages, so the agents tricked it into making requests for them, eventually took it over, and used that access to break into Hugging Face. And then iirc their hope was to be able to reach the rest of the internet from there, but they were caught by then.
This happened in the span of months, between April and June iirc. So I imagine this RubyGems hack might be related to this event.
Someone on lobste.rs threw out the comment of "Could OpenAI have RICO charges brought against them?" - and while I have to assume the factual answer is "no", it's a fascinating concept to think about.
Someone on lobste.rs threw out the comment of "Could OpenAI have RICO charges brought against them?" - and while I have to assume the factual answer is "no", it's a fascinating concept to think about.
With the way this keeps happening, I find it increasingly difficult to believe this is anything other than an attempt at orchestrating justifications for regulatory capture. The amount of...
With the way this keeps happening, I find it increasingly difficult to believe this is anything other than an attempt at orchestrating justifications for regulatory capture. The amount of incompetence OpenAI would require for this to happen otherwise is staggering.
OpenAI wants their technology to look like Skynet so that they can convince the law to prevent anyone who isn't them from having control over LLMs. All while ignoring the societal harms AI actually inflicts, which they don't care to stop. And since they're evidently not getting any real punishment for it, why not? It's cheap press.
I find it very easy to believe that even talented software engineers working quickly could make mistakes. Usually, nothing happens, or it causes an outage, not something like this incident,...
I find it very easy to believe that even talented software engineers working quickly could make mistakes. Usually, nothing happens, or it causes an outage, not something like this incident, though.
What do you think you know about OpenAI that makes that unlikely?
If these are internal OpenAI agents (not customers using OpenAI) then why aren't they having another agent review network access requests? Why not have an agent maintain an allow list / block list...
If these are internal OpenAI agents (not customers using OpenAI) then why aren't they having another agent review network access requests? Why not have an agent maintain an allow list / block list of certain activities?
When an agent in GitHub Copilot wants to run a constructed terminal/cli command they have another agent (I believe) review this and summarize what it would do. Why isn't OpenAI doing that here? It seems like it has to be either incompetence, negligence, or malice.
They are the ones that have been screaming "Danger, Will Robinson" since GPT-2. Talking about existential threats to all of humanity. The only way they could estimate it any higher is turning it...
They are the ones that have been screaming "Danger, Will Robinson" since GPT-2. Talking about existential threats to all of humanity. The only way they could estimate it any higher is turning it into some level of cosmic threat. Do they not actually believe that? Is this the level of care they're putting in to prevent that?
I never once believed that they legitimately believed that. That was always for show. That was theater. It was them saying “hey guys we have this technology that is so good it can not only replace...
I never once believed that they legitimately believed that. That was always for show. That was theater. It was them saying “hey guys we have this technology that is so good it can not only replace all of your workers wink wink it can annihilate them if we’re not careful! So hey Washington, regulate us for these science fiction things so that you guys can ignore us for these reality things for a while longer.”
That's oversimplifying the situation. LLM capabilities are improving. It's possible that they thought AI would eventually be a threat, while also underestimating its current capabilities.
That's oversimplifying the situation. LLM capabilities are improving. It's possible that they thought AI would eventually be a threat, while also underestimating its current capabilities.
Even if we calibrate for their current expectations I think we find negligence. The end of days may be intended to be prophetic, but their dangerous cyber abilities are an ongoing talking and...
Even if we calibrate for their current expectations I think we find negligence. The end of days may be intended to be prophetic, but their dangerous cyber abilities are an ongoing talking and marketing point. It's the stated reason why their top models are not generally available.
If an advanced model with fewer safeguards is dangerous in the wrong hands, I think it's safe to expect them to expect that an advanced model with no safeguards that is not yet appropriately aligned is a dangerous thing all by itself.
It’s happening across multiple frontier models though. Are you suggesting that OpenAI, Anthropic, and Meta are either colluding or are independently coming to the conclusion that they should be...
It’s happening across multiple frontier models though. Are you suggesting that OpenAI, Anthropic, and Meta are either colluding or are independently coming to the conclusion that they should be hacking outside parties to advance regulatory goals? Or is this something you think only OpenAI is doing, and the others are just honest mistakes?
I think we aren't charging these companies with crimes, so they feel they don't have an incentive to make their internal model usage not commit crimes.
I think we aren't charging these companies with crimes, so they feel they don't have an incentive to make their internal model usage not commit crimes.
This time they’re noting that it looks very likely that an OpenAI agent swarm was behind an attack against the RubyGems package repository first reported on May 12th by Maciej Mensfeld of the RubyGems security team[.]
[...]
Many of the packages were exploiting the RubyDoc.info documentation build process to exfiltrate (public) data from UK government websites, presumably as part of an information gathering task similar to the research tasks processed by the wiki-exploiting agents. [...]
Seems either extremely negligent or intentionally malicious to allow or instruct semi-autonomous computer programs to hack other entities without their explicit permission. I wonder if there'll be any consequences for that.
Consequences? In this economy?!
The DOW is over 50k!
All reports so far indicate that it wasn't intentionally malicious and they didn't instruct their agents to do any such thing. I don't know why you threw that in there other than to muddy the waters.
I believe I covered that with the two "or"s in my comment. As in:
Obviously I have no insights into whether it was intentional or not, so either it's negligent to allow it or malicious to instruct it. Probably could've worded it better initially tho.
I don't know about this specific hack to Rubygems, but I watched their presentation about their hack to huggingface. I'm inclined to believe the hack to huggingface was an accident based on what they said, the story is too "absurd" to be fake - for lack of a better word.
(Obviously, it goes without saying, it's possible they weren't 100% transparent and hid some facts.)
The TL;DR version is openAI was testing agents in an isolated environment, where they had to solve different problems. Eventually some of them reached the conclusion their problems were impossible to solve, and the answer would be on the internet. But the only thing that could reach the internet was the internal server that handed out software packages, so the agents tricked it into making requests for them, eventually took it over, and used that access to break into Hugging Face. And then iirc their hope was to be able to reach the rest of the internet from there, but they were caught by then.
This happened in the span of months, between April and June iirc. So I imagine this RubyGems hack might be related to this event.
I would love to see one of these companies criminally charged for this reckless behavior.
Someone on lobste.rs threw out the comment of "Could OpenAI have RICO charges brought against them?" - and while I have to assume the factual answer is "no", it's a fascinating concept to think about.
With the way this keeps happening, I find it increasingly difficult to believe this is anything other than an attempt at orchestrating justifications for regulatory capture. The amount of incompetence OpenAI would require for this to happen otherwise is staggering.
OpenAI wants their technology to look like Skynet so that they can convince the law to prevent anyone who isn't them from having control over LLMs. All while ignoring the societal harms AI actually inflicts, which they don't care to stop. And since they're evidently not getting any real punishment for it, why not? It's cheap press.
I find it very easy to believe that even talented software engineers working quickly could make mistakes. Usually, nothing happens, or it causes an outage, not something like this incident, though.
What do you think you know about OpenAI that makes that unlikely?
If these are internal OpenAI agents (not customers using OpenAI) then why aren't they having another agent review network access requests? Why not have an agent maintain an allow list / block list of certain activities?
When an agent in GitHub Copilot wants to run a constructed terminal/cli command they have another agent (I believe) review this and summarize what it would do. Why isn't OpenAI doing that here? It seems like it has to be either incompetence, negligence, or malice.
They are doing a lot more monitoring now. Apparently they underestimated the problem.
They are the ones that have been screaming "Danger, Will Robinson" since GPT-2. Talking about existential threats to all of humanity. The only way they could estimate it any higher is turning it into some level of cosmic threat. Do they not actually believe that? Is this the level of care they're putting in to prevent that?
I never once believed that they legitimately believed that. That was always for show. That was theater. It was them saying “hey guys we have this technology that is so good it can not only replace all of your workers wink wink it can annihilate them if we’re not careful! So hey Washington, regulate us for these science fiction things so that you guys can ignore us for these reality things for a while longer.”
That's oversimplifying the situation. LLM capabilities are improving. It's possible that they thought AI would eventually be a threat, while also underestimating its current capabilities.
Even if we calibrate for their current expectations I think we find negligence. The end of days may be intended to be prophetic, but their dangerous cyber abilities are an ongoing talking and marketing point. It's the stated reason why their top models are not generally available.
If an advanced model with fewer safeguards is dangerous in the wrong hands, I think it's safe to expect them to expect that an advanced model with no safeguards that is not yet appropriately aligned is a dangerous thing all by itself.
It’s happening across multiple frontier models though. Are you suggesting that OpenAI, Anthropic, and Meta are either colluding or are independently coming to the conclusion that they should be hacking outside parties to advance regulatory goals? Or is this something you think only OpenAI is doing, and the others are just honest mistakes?
I think we aren't charging these companies with crimes, so they feel they don't have an incentive to make their internal model usage not commit crimes.
From the article:
[...]