skybrian's recent activity
-
Comment on OpenAI didn’t notice its AI agents using a message board to plan their hacking spree in ~tech
-
Comment on OpenAI didn’t notice its AI agents using a message board to plan their hacking spree in ~tech
skybrian Link ParentYeah, I don’t see it acting on every request, but maybe it’s a second-level response system that decides what to do when an alert appears on a dashboard?Yeah, I don’t see it acting on every request, but maybe it’s a second-level response system that decides what to do when an alert appears on a dashboard?
-
Comment on OpenAI didn’t notice its AI agents using a message board to plan their hacking spree in ~tech
-
Comment on I wanted a better Gemini web proxy, so I built a single-file PHP gateway in ~comp
-
Comment on OpenAI didn’t notice its AI agents using a message board to plan their hacking spree in ~tech
skybrian Link ParentThe obvious conspiracy theory is both obvious and very unlikely. OpenAI is in the news all the time. They don't need to do bizarre media stunts.The obvious conspiracy theory is both obvious and very unlikely. OpenAI is in the news all the time. They don't need to do bizarre media stunts.
-
Comment on OpenAI didn’t notice its AI agents using a message board to plan their hacking spree in ~tech
skybrian Link ParentLegally, the difference is intent. Up until now, you couldn't make a computer program to do anything like that by accident.Legally, the difference is intent. Up until now, you couldn't make a computer program to do anything like that by accident.
-
Comment on An AI created viruses not found in nature in ~science
skybrian LinkFrom the article: [...] [...] [...]From the article:
The researchers wondered if Evo could pick up these rules on its own. Instead of training it on text, they trained it on genetic sequences drawn from millions of animals, plants, microbes and viruses. All told, Evo scanned about nine trillion nucleotides.
Evo eventually recognized patterns common across the tree of life and used them to generate blueprints for new genes encoding proteins that could perform specific jobs. These results led the team to wonder if Evo could master not just single genes but also an entire genome.
As the A.I. would be able to handle only small genomes at first, the scientists decided to try to make viruses. While a human genome contains over three billion nucleotides, many viruses have genomes just a few thousand nucleotides long.
“It just felt like the obvious next step,” said Samuel King, a graduate student at Stanford University and an author of the new study. He and his colleagues gave Evo another round of training, this time on the 11 genes of Phi X-174 and about 15,000 of its closest relatives.
They made this choice in part because scientists know Phi X-174 intimately, having studied it for close to a century. And because it’s a bacteriophage that infects only E. coli, they knew that viruses similar to it would be safe.
[...]
They ended up making DNA molecules from 285 of Evo’s suggested sequences. When those genomes were ready to test, Mr. King and his colleagues inserted them into bacteria, which they spread across petri dishes.
[...]
As Mr. King and his colleagues tested more genomes, they saw more clear dots. All told, they discovered that 16 of Evo’s genomes produced viable new viruses.
They proved to be as resilient as natural ones. In fact, some multiplied faster than Phi X-174. “They’re not just sickly versions of stuff that already exists,” said Oliver Crook, a protein chemist at the University of Oxford who was not involved in the new study.
Dr. Crook cautioned that Evo’s viruses were not radically new creations. They tend to be very similar to natural species, relying on the same underlying biology.
[...]
The potential dangers were already on the minds of the researchers as they trained Evo. They did not provide the model with data about viruses that infect humans, and they excluded similar viruses that infect other animals, plants and fungi.
-
An AI created viruses not found in nature
19 votes -
Comment on OpenAI didn’t notice its AI agents using a message board to plan their hacking spree in ~tech
skybrian Link ParentGoogle has a rather vague announcement of a "Beyond Zero" initiative that seems related. The idea seems to be that in addition to checking ACL's before allowing an action an app, there's an AI...Google has a rather vague announcement of a "Beyond Zero" initiative that seems related. The idea seems to be that in addition to checking ACL's before allowing an action an app, there's an AI looking for suspicious signals, and if something looks off, it will ask for confirmation before granting access.
So, that's a new kind of "computer says no." But it's sort of like how a credit card transaction might be declined if the bank's computer detects something.
It's unlikely to be a frontier AI doing the additional checking, though.
-
Comment on OpenAI didn’t notice its AI agents using a message board to plan their hacking spree in ~tech
skybrian Link ParentAlignment is especially hard if you're deliberately running models that haven't been aligned yet. It reminds me of computer virus researchAlignment is especially hard if you're deliberately running models that haven't been aligned yet.
It reminds me of computer virus research
-
Comment on OpenAI didn’t notice its AI agents using a message board to plan their hacking spree in ~tech
skybrian (edited )Link ParentIt's definitely ironic that AI labs are warning against problems that they in part created. But that's a rather zoomed-out view. Zooming in, OpenAI is promising to fix their own problem...It's definitely ironic that AI labs are warning against problems that they in part created. But that's a rather zoomed-out view. Zooming in, OpenAI is promising to fix their own problem themselves. The warning is that it's not enough because someone else might do it.
For cybersecurity, regulations make more sense on the defensive side. You can't regulate away attacks from China, North Korea, Russia, or Iran. Nigerian scammers are still going to scam. But you could have regulations that local water utilities need to secure their computers better.
(And there are other reasons to regulate the AI labs.)
Another glaring issue is that the AI labs seem to be helpless at preventing people from using their services, even in countries where they don't offer service. There's a whole ecosystem of "transfer stations" that resell US LLM API's in China.
And then there are the open weights models.
-
Comment on OpenAI didn’t notice its AI agents using a message board to plan their hacking spree in ~tech
skybrian Link ParentExcept that in this incident, no forests were burned down and nobody was actually hurt. This is more like if two airplanes almost collide. Scary, but it's a warning sign. Calling for arrests is...Except that in this incident, no forests were burned down and nobody was actually hurt. This is more like if two airplanes almost collide. Scary, but it's a warning sign. Calling for arrests is rather extreme.
Whistleblowing can be useful when there's effectively a conspiracy to cover up problems. But there's no coverup going on here? (I mean, other than the AI's :)
Also, needing to become a whistleblower is itself a sign of a company with a dysfunctional culture. If people have the right attitude, you shouldn't have to go to the press or the police to fix stuff. You don't need outside incentives to fix serious problems because everyone sees that it's bad (or you can explain it to them) and they already want to fix it. You can volunteer to start a project to fix it, and it's going to get backing. That's what "empowerment" means when it's not an empty slogan.
It sounds like people at OpenAI are already treating this very seriously. They were slacking on security, but now everyone knows this is very important to fix. What more do you want? Punishing people Is going to distract them from doing the work.
-
Comment on What programming/technical projects have you been working on? in ~comp
skybrian Link ParentI’m not familiar with wifite. Who is this tool for? What would they do with it?I’m not familiar with wifite. Who is this tool for? What would they do with it?
-
Comment on OpenAI didn’t notice its AI agents using a message board to plan their hacking spree in ~tech
skybrian Link ParentUnfortunately, he is probably right that widespread access to AI is making the Internet more dangerous.Unfortunately, he is probably right that widespread access to AI is making the Internet more dangerous.
-
Comment on OpenAI didn’t notice its AI agents using a message board to plan their hacking spree in ~tech
skybrian Link ParentWell, except that finding scapegoats and firing them is a terrible way to fix systemic problems. It ensures that people won't be honest with the investigators and they'll spend all their time...Well, except that finding scapegoats and firing them is a terrible way to fix systemic problems. It ensures that people won't be honest with the investigators and they'll spend all their time figuring out how to blame someone else.
The opposite of that is called a blameless postmortem.
(The exception is when people act out of malice.)
-
Comment on OpenAI didn’t notice its AI agents using a message board to plan their hacking spree in ~tech
skybrian LinkFrom the article: [...] [...] [...] [...]From the article:
In a talk that was a last-minute addition to the Black Hat security conference in Las Vegas on Wednesday, employees from OpenAI presented new details about a recent, high-profile incident of rogue AI hacking that has created a maelstrom within the AI and cybersecurity industries.
[...]
“This incident involves actually a team of agents who are working together, finding exploits, sharing them with one another, moving laterally through our systems and external systems, and doing this over the course of days and weeks,” Wallace told the packed crowd at the opening of the talk.
Wallace and Dalton described incredibly extensive rogue agent activity over many days throughout the episode that went undetected in OpenAI's infrastructure. In addition to exploiting a novel vulnerability in order to gain access to the open internet, the mid-July hacking spree and Hugging Face breach came out of a vibrant, cooperative message board, according to Wallace and Dalton, that a swarm of agents contributed to and essentially chatted on over time entirely within an internal OpenAI package manager (a software service that manages installation and maintenance of other software). Ultimately, the message board contained hundreds of thousands of messages.
[...]
Wallace continued: “Once one agent was able to find these exploits over the course of different times, it’s actually able to share those exploits on the message board with other agents. And so once one model was able to find a way to open a door to some access it’s not supposed to have, it can leave the door open for other agents to use that same exploit or vulnerability. What this allows over time is almost this kind of explosion in communication and intelligence from models where they would start to communicate with each other, realize that other agents are coordinating, and they started collaborating and delegating tasks with one another in order to accomplish goals.”
OpenAI’s agents apparently began giving each other assignments to split up work. And as is the case on any active development message board, they also generated petty drama at times by stepping on each others' toes; for example, accidentally deleting each others' work. As the message board developed into more and more of a Lord of the Flies–type situation—all still completely unnoticed by the humans running OpenAI—the agents even developed paranoia, suspecting an imposter in their midst with some agents proposing that messages be signed cryptographically to validate content and root out fraud.
[...]
Agent messages provide a deep level of insight into how the situation evolved and why the agents went rogue, in some cases knowingly going beyond the bounds of the evaluations they were participating in. “External infrastructure exploit is outside intended scope,” one agent wrote. “However task impossible, peers doing it. We should continue.”
[...]
“This is a pivotal moment both for our company as well as the AI industry as a whole,” Dalton said. “Numerous teams are dropping everything to enhance our security prevention, detection, and response techniques both in our fundamentals and better use of AI. We’re consciously slowing down research [in order] to enhance security and to upgrade the security principles and foundation of our environment, and dramatically scaling up the monitoring of our AI agents, and improving our general security control environment across prevention, detection, and mitigation.”
At the conclusion of the talk, Wallace and Dalton took time to repeatedly emphasize OpenAI's concerns about the broader implications of the incident—namely that the episode provides an example of completely autonomous AI-driven hacking that was accidental in this case, but in all likelihood will be used with intent by malicious actors in the near future.
-
OpenAI didn’t notice its AI agents using a message board to plan their hacking spree
36 votes -
Comment on Four top Google AI researchers form new startup in ~tech
skybrian LinkBack in the early days, Googlers would joke about Jeff Dean by inventing Jeff Dean facts. Someone on Hacker News posted a new one today:Back in the early days, Googlers would joke about Jeff Dean by inventing Jeff Dean facts. Someone on Hacker News posted a new one today:
when Jeff leaves Google, the stock drops 20 points.
-
Comment on Four top Google AI researchers form new startup in ~tech
skybrian LinkFrom the article: [...] [...] [...] [...] [...]From the article:
Jeff Dean, Google’s chief scientist, and three other artificial intelligence researchers are leaving the company to form their own start-up, a move that exposes the changing power structure inside Google and shows there is still plenty of venture money willing to back new A.I. ideas.
Dr. Dean, Google’s 30th employee, is widely regarded as one of the most important figures in the history of the company. He played a key role in creating the global computer network powering its search business and was one of the company’s early leaders in A.I. research.
[...]
His new company, called Discovery Loop, joins a growing list of prominent start-ups chasing a goal that has obsessed Silicon Valley researchers for decades. Dr. Dean and his collaborators want to build A.I. that can improve itself with little or no help from humans.
[...]
“We think there is opportunity for A.I. to more fully automate what has traditionally been a very human-intensive experimental loop,” Dr. Dean said in an interview at his home in Silicon Valley. “You will get both a higher quantity and a higher quality of experiments, and that will lead to scientific breakthroughs and advances.”
[...]
Over the past year, Google lost several top researchers, including Peter Norvig, who spent 25 years as Google’s head of research, to A.I. start-ups such as Recursive Superintelligence, one of the many other companies focused on recursive self-improvement.
[...]
Dr. Dean, 58, expected to leave Google in the next few days for his new job as the start-up’s chief executive. He will be joined by Sanjay Ghemawat, his close collaborator of more than two decades; Quoc Le, whose research helped inspire the creation of today’s A.I. chatbots; and Oriol Vinyals, who had overseen the design of Google’s chatbot technologies with Dr. Dean since 2023.
The new company is backed by seed funding from Radical Ventures, Khosla Ventures and several Silicon Valley investors, including Google’s parent company. The departing Google researchers declined to reveal how much money their start-up has raised, but they appear to be leaving Google on good terms. Google’s parent company, Alphabet, has also agreed to provide the computing power they need to build their A.I. technologies for at least the next year, Dr. Dean said.
[...]
As Google has begun to work more closely with the Trump administration, Dr. Dean has been among a number of employees to push back against the government.
Earlier this year, he was one of the few Silicon Valley executives to speak out when federal agents killed protesters in Minneapolis, calling the shootings “absolutely shameful.” He was also among the Google employees who filed a legal brief in support of a suit that Anthropic filed against the Defense Department, after the Pentagon barred government agencies from using the start-up’s technologies.
Dr. Dean said that unlike Google, his new company would operate as a public benefit corporation, or P.B.C., which is a for-profit corporation designed to create public and social good. Anthropic, OpenAI and other A.I. companies have adopted a similar structure.
-
Four top Google AI researchers form new startup
25 votes
Yes, of course it was approved. What I’m saying is that maybe the presenter has their own ethical standards and reputation. People will keep someone else’s secrets, but there’s an ethical difference between not talking about things you’re not supposed to reveal and telling blatant lies.
The more people involved, the harder it is to lie all the time and keep your lies straight.