skybrian's recent activity

  1. Comment on OpenAI didn’t notice its AI agents using a message board to plan their hacking spree in ~tech

    skybrian
    Link Parent
    Alignment is especially hard if you're deliberately running models that haven't been aligned yet. It reminds me of computer virus research

    Alignment is especially hard if you're deliberately running models that haven't been aligned yet.

    It reminds me of computer virus research

  2. Comment on OpenAI didn’t notice its AI agents using a message board to plan their hacking spree in ~tech

    skybrian
    (edited )
    Link Parent
    It's definitely ironic that AI labs are warning against problems that they in part created. But that's a rather zoomed-out view. Zooming in, OpenAI is promising to fix their own problem...

    It's definitely ironic that AI labs are warning against problems that they in part created. But that's a rather zoomed-out view. Zooming in, OpenAI is promising to fix their own problem themselves. The warning is that it's not enough because someone else might do it.

    For cybersecurity, regulations make more sense on the defensive side. You can't regulate away attacks from China, North Korea, Russia, or Iran. Nigerian scammers are still going to scam. But you could have regulations that local water utilities need to secure their computers better.

    (And there are other reasons to regulate the AI labs.)

    Another glaring issue is that the AI labs seem to be helpless at preventing people from using their services, even in countries where they don't offer service. There's a whole ecosystem of "transfer stations" that resell US LLM API's in China.

    And then there are the open weights models.

  3. Comment on OpenAI didn’t notice its AI agents using a message board to plan their hacking spree in ~tech

    skybrian
    Link Parent
    Except that in this incident, no forests were burned down and nobody was actually hurt. This is more like if two airplanes almost collide. Scary, but it's a warning sign. Calling for arrests is...

    Except that in this incident, no forests were burned down and nobody was actually hurt. This is more like if two airplanes almost collide. Scary, but it's a warning sign. Calling for arrests is rather extreme.

    Whistleblowing can be useful when there's effectively a conspiracy to cover up problems. But there's no coverup going on here? (I mean, other than the AI's :)

    Also, needing to become a whistleblower is itself a sign of a company with a dysfunctional culture. If people have the right attitude, you shouldn't have to go to the press or the police to fix stuff. You don't need outside incentives to fix serious problems because everyone sees that it's bad (or you can explain it to them) and they already want to fix it. You can volunteer to start a project to fix it, and it's going to get backing. That's what "empowerment" means when it's not an empty slogan.

    It sounds like people at OpenAI are already treating this very seriously. They were slacking on security, but now everyone knows this is very important to fix. What more do you want? Punishing people Is going to distract them from doing the work.

    3 votes

  4. Comment on What programming/technical projects have you been working on? in ~comp

    skybrian
    Link Parent
    I’m not familiar with wifite. Who is this tool for? What would they do with it?

    I’m not familiar with wifite. Who is this tool for? What would they do with it?

  5. Comment on OpenAI didn’t notice its AI agents using a message board to plan their hacking spree in ~tech

    skybrian
    Link Parent
    Unfortunately, he is probably right that widespread access to AI is making the Internet more dangerous.

    Unfortunately, he is probably right that widespread access to AI is making the Internet more dangerous.

    2 votes

  6. Comment on OpenAI didn’t notice its AI agents using a message board to plan their hacking spree in ~tech

    skybrian
    Link Parent
    Well, except that finding scapegoats and firing them is a terrible way to fix systemic problems. It ensures that people won't be honest with the investigators and they'll spend all their time...

    Well, except that finding scapegoats and firing them is a terrible way to fix systemic problems. It ensures that people won't be honest with the investigators and they'll spend all their time figuring out how to blame someone else.

    The opposite of that is called a blameless postmortem.

    (The exception is when people act out of malice.)

    16 votes

  7. Comment on OpenAI didn’t notice its AI agents using a message board to plan their hacking spree in ~tech

    skybrian
    Link
    From the article: [...] [...] [...] [...]

    From the article:

    In a talk that was a last-minute addition to the Black Hat security conference in Las Vegas on Wednesday, employees from OpenAI presented new details about a recent, high-profile incident of rogue AI hacking that has created a maelstrom within the AI and cybersecurity industries.

    [...]

    “This incident involves actually a team of agents who are working together, finding exploits, sharing them with one another, moving laterally through our systems and external systems, and doing this over the course of days and weeks,” Wallace told the packed crowd at the opening of the talk.

    Wallace and Dalton described incredibly extensive rogue agent activity over many days throughout the episode that went undetected in OpenAI's infrastructure. In addition to exploiting a novel vulnerability in order to gain access to the open internet, the mid-July hacking spree and Hugging Face breach came out of a vibrant, cooperative message board, according to Wallace and Dalton, that a swarm of agents contributed to and essentially chatted on over time entirely within an internal OpenAI package manager (a software service that manages installation and maintenance of other software). Ultimately, the message board contained hundreds of thousands of messages.

    [...]

    Wallace continued: “Once one agent was able to find these exploits over the course of different times, it’s actually able to share those exploits on the message board with other agents. And so once one model was able to find a way to open a door to some access it’s not supposed to have, it can leave the door open for other agents to use that same exploit or vulnerability. What this allows over time is almost this kind of explosion in communication and intelligence from models where they would start to communicate with each other, realize that other agents are coordinating, and they started collaborating and delegating tasks with one another in order to accomplish goals.”

    OpenAI’s agents apparently began giving each other assignments to split up work. And as is the case on any active development message board, they also generated petty drama at times by stepping on each others' toes; for example, accidentally deleting each others' work. As the message board developed into more and more of a Lord of the Flies–type situation—all still completely unnoticed by the humans running OpenAI—the agents even developed paranoia, suspecting an imposter in their midst with some agents proposing that messages be signed cryptographically to validate content and root out fraud.

    [...]

    Agent messages provide a deep level of insight into how the situation evolved and why the agents went rogue, in some cases knowingly going beyond the bounds of the evaluations they were participating in. “External infrastructure exploit is outside intended scope,” one agent wrote. “However task impossible, peers doing it. We should continue.”

    [...]

    “This is a pivotal moment both for our company as well as the AI industry as a whole,” Dalton said. “Numerous teams are dropping everything to enhance our security prevention, detection, and response techniques both in our fundamentals and better use of AI. We’re consciously slowing down research [in order] to enhance security and to upgrade the security principles and foundation of our environment, and dramatically scaling up the monitoring of our AI agents, and improving our general security control environment across prevention, detection, and mitigation.”

    At the conclusion of the talk, Wallace and Dalton took time to repeatedly emphasize OpenAI's concerns about the broader implications of the incident—namely that the episode provides an example of completely autonomous AI-driven hacking that was accidental in this case, but in all likelihood will be used with intent by malicious actors in the near future.

    7 votes

  9. Comment on Four top Google AI researchers form new startup in ~tech

    skybrian
    Link
    Back in the early days, Googlers would joke about Jeff Dean by inventing Jeff Dean facts. Someone on Hacker News posted a new one today:

    Back in the early days, Googlers would joke about Jeff Dean by inventing Jeff Dean facts. Someone on Hacker News posted a new one today:

    when Jeff leaves Google, the stock drops 20 points.

    12 votes

  10. Comment on Four top Google AI researchers form new startup in ~tech

    skybrian
    Link
    From the article: [...] [...] [...] [...] [...]

    From the article:

    Jeff Dean, Google’s chief scientist, and three other artificial intelligence researchers are leaving the company to form their own start-up, a move that exposes the changing power structure inside Google and shows there is still plenty of venture money willing to back new A.I. ideas.

    Dr. Dean, Google’s 30th employee, is widely regarded as one of the most important figures in the history of the company. He played a key role in creating the global computer network powering its search business and was one of the company’s early leaders in A.I. research.

    [...]

    His new company, called Discovery Loop, joins a growing list of prominent start-ups chasing a goal that has obsessed Silicon Valley researchers for decades. Dr. Dean and his collaborators want to build A.I. that can improve itself with little or no help from humans.

    [...]

    “We think there is opportunity for A.I. to more fully automate what has traditionally been a very human-intensive experimental loop,” Dr. Dean said in an interview at his home in Silicon Valley. “You will get both a higher quantity and a higher quality of experiments, and that will lead to scientific breakthroughs and advances.”

    [...]

    Over the past year, Google lost several top researchers, including Peter Norvig, who spent 25 years as Google’s head of research, to A.I. start-ups such as Recursive Superintelligence, one of the many other companies focused on recursive self-improvement.

    [...]

    Dr. Dean, 58, expected to leave Google in the next few days for his new job as the start-up’s chief executive. He will be joined by Sanjay Ghemawat, his close collaborator of more than two decades; Quoc Le, whose research helped inspire the creation of today’s A.I. chatbots; and Oriol Vinyals, who had overseen the design of Google’s chatbot technologies with Dr. Dean since 2023.

    The new company is backed by seed funding from Radical Ventures, Khosla Ventures and several Silicon Valley investors, including Google’s parent company. The departing Google researchers declined to reveal how much money their start-up has raised, but they appear to be leaving Google on good terms. Google’s parent company, Alphabet, has also agreed to provide the computing power they need to build their A.I. technologies for at least the next year, Dr. Dean said.

    [...]

    As Google has begun to work more closely with the Trump administration, Dr. Dean has been among a number of employees to push back against the government.

    Earlier this year, he was one of the few Silicon Valley executives to speak out when federal agents killed protesters in Minneapolis, calling the shootings “absolutely shameful.” He was also among the Google employees who filed a legal brief in support of a suit that Anthropic filed against the Defense Department, after the Pentagon barred government agencies from using the start-up’s technologies.

    Dr. Dean said that unlike Google, his new company would operate as a public benefit corporation, or P.B.C., which is a for-profit corporation designed to create public and social good. Anthropic, OpenAI and other A.I. companies have adopted a similar structure.

    12 votes

  12. Comment on A new book details Israel’s destruction of Palestinian life in Gaza in ~books

    skybrian
    Link
    https://archive.is/IZ7ew From the article: [...] [...] [...]

    https://archive.is/IZ7ew

    From the article:

    A new book by the British Israeli architect and political activist Eyal Weizman tries to fill that void by telling the story of Israel’s systematic destruction of Gaza street by street, farm by farm, and hospital by hospital. What sets Ungrounding: The Architecture of Genocide apart from this polarized discourse is its determined focus on the physical and its analytical tone. The evidence, collected by Weizman and his team from audios, videos, and interviews and subjected to rigorous analysis, is overwhelming. Their efforts produced the “Cartography of Genocide,” an interactive and constantly evolving digital map that details “an organized and designed campaign to destroy Palestinian life in Gaza.” By the end of the book, the question is less whether Israel’s actions should be regarded as genocide and more what to make of a genocidal process that seeks to not only remove a people but to fundamentally and permanently transform the land.

    [...]

    But Weizman’s goals here are not strictly academic. His prose shimmers with barely contained rage at the horrors that his team observed, that Israelis deny, and that the media is barred from covering on the ground. Parts of the book read like a murder mystery, as he explains the methods and evidence that his team used to painstakingly reconstruct specific war crimes or events, showing how Israeli spokespeople produced and deceptively edited misleading or unsupported claims. Familiar horror stories, such as the destruction of the Al-Ahli Hospital or the killing of young Hind Rajab, take on new life in his clinical, precise dispatches. There can be something tedious, even actively perfidious, about how Israel’s defenders force analysts deep into the weeds, arguing over microscopic details when the big picture is clear. But Weizman does it well, and the forensic demonstration of what happened at the micro level provides essential support for the credibility of the larger narrative.

    [...]

    Israel had long declared buffer zones where any Palestinian would be killed. It relentlessly expanded those zones as the invasion proceeded. Palestinians never really knew where the demarcation line ran. At any moment, they might wander into an unrecognized forbidden zone and be killed on sight. By July 2025, this buffer zone covered 82 percent of Gaza’s territory, according to Weizman. Gaza’s population was systematically herded into increasingly smaller “safe zones” that were anything but safe, forced to live on the sand dunes and in areas lacking even the most basic necessities of life. The intermittent closure of Gaza to aid reinforced the misery: “The humanitarian zone on the dunes is where conditions of life were calculated to keep people barely alive, to make conditions so unlivable that Palestinians would want to leave the moment the borders opened.”

    Meanwhile, in the territories emptied of civilians, U.S.-provided bombs “landed on residential and commercial tower blocks, shopping centers, schools, mosques, bakeries, banks, the beach promenade, hospitals, the legislative council, bookshops, publishing houses, libraries, hundreds of educational faculties, universities, food warehouses, restaurants.” Weizman demonstrates how the ungrounding of the buffer zones created environmental catastrophe, poisoning the soil for generations. “Pipework in the shallow subsoil was cut by the bulldozers and wastewater bled out. Without fuel, sewage facilities spilled millions of liters onto the exposed surface which from there made their way into the subsoil,” he writes. By October 2024, almost 4,000 greenhouses were destroyed, orchards were bulldozed, and almost all cows, sheep and poultry were eliminated. By April 2025, more than 80 percent of Gaza’s farmland had been destroyed, according to the book.

    [...]

    But what sets Weizman’s book apart from other depictions of Gaza’s horrors is that he takes Israeli concerns about tunnels and Hamas insurgent practices seriously. Much of the book, in fact, is taken up with fascinating depictions of those tunnels—how they were made, how they have been used, and their effects on Gazan society and economics. Along the way, he unpacks and dismisses a fair amount of Israeli propaganda. The tunnels were never an underground metro with the elaborate furnishings depicted in Israeli graphics. If there had been such an interconnected system, after all, it would have been exposed the first time that Israeli soldiers entered a tunnel—and would have succumbed to various Israeli tactics such as seawater flooding, poison gas, or carpet bombing along the supposed route.

    7 votes

  14. Comment on An agent workspace built on Cloudflare workers in ~comp

    skybrian
    Link
    From the README: From Kenton Varda's Twitter post:

    From the README:

    Cloudflare OS is an "operating system" for AI productivity originally developed for use inside Cloudflare. A large portion of Cloudflare's workforce -- from engineering to sales and everything in between -- uses Cloudflare OS every day to help them do their jobs.

    This is not a traditional computer operating system. We use the term "operating system" in two senses:

    • An operating system for the company to be productive with AI, in a way that is safe, so that the security team can sleep at night.

    • An operating system for AI workloads, analogous to the sense in which a traditional operating system manages compute workloads.

    From Kenton Varda's Twitter post:

    This is a remake of Sandstorm[.]io, my startup from 10 years ago, except this time built on Cloudflare Workers (the platform I've spent the last 9 years building) and deeply leveraging AI. This is more or less the culmination of my secret 10-year master plan.

    This is a full-on personal app vibe coding platform, in which the sandbox is so secure that you can pretty much go wild -- the AI cannot introduce a significant security bug. We believe a company's security team can feel comfortable giving non-technical users permission to vibe code and then sleep soundly at night.

    How is that possible? It's the Sandstorm security model, revisited. [...]

  16. Comment on Inside Google’s $200bn Wall Street finance machine for Anthropic in ~finance

    skybrian
    (edited )
    Link Parent
    You might be underestimating how deeply AI is being embedded into some companies. For example, here’s what Cloudflare is doing. The AI labs have poor uptimes, but that means they will switch to a...

    You might be underestimating how deeply AI is being embedded into some companies. For example, here’s what Cloudflare is doing.

    The AI labs have poor uptimes, but that means they will switch to a different LLM provider rather than do without.

    I imagine it’s going to be an expected utility like email and Internet access and cell phone service. Some banking services will degrade or stop working altogether if their AI goes down and they have to revert to manual procedures.

    But I don’t expect bailouts anytime soon because switching is pretty easy. It will be a temporary disruption like an airline going bankrupt. Think of a data center like an airplane that some other airline could lease.

    Also, the government doesn’t particularly like any AI companies and neither do the people. It’s not like the auto industry where factory workers have a lot of clout.

    3 votes

  17. Comment on Inside Google’s $200bn Wall Street finance machine for Anthropic in ~finance

    skybrian
    Link
    https://archive.is/mC6hM From the article: [...] [...] [...] [...]

    https://archive.is/mC6hM

    From the article:

    Google has assembled one of the largest infrastructure financing programmes in history to supply more than $150bn of artificial intelligence chips to Anthropic.

    Surging demand from Anthropic, in which Google is an investor, has led the Big Tech company to orchestrate a sprawling operation to supply its chips to the start-up, according to people involved in the project and corporate filings reviewed by the FT.

    The effort brings together Google, Broadcom, Apollo, Blackstone, Morgan Stanley and a slew of crypto miners in a web of transactions that stretches from chip manufacturing to data centre development.

    At the centre of the project are Google’s tensor processing units, or TPUs — AI chips it has co-developed with Broadcom since 2016. Once used largely inside Google’s own data centres, the chips have begun to be sold externally, challenging Nvidia’s dominance of the AI processor market.

    [...]

    To support the relentless surge in demand for the AI chips, Google, Broadcom and Wall Street investors have each taken on different pieces of the financial risk.

    Google guarantees the data centres. Broadcom commits to buying the chips and helps finance them. Apollo and Blackstone provide much of the private-credit capital that purchases the hardware before leasing it to Anthropic.

    “This is each of us putting our balance sheet to work,” said a Google executive involved in the effort. “We’re doing it on the data centre side, [Broadcom’s] doing it on the chip side.”The web of contracts underpinning these arrangements adds up to about $200bn, with roughly four-fifths tied to the chips themselves, making it one of the largest infrastructure financings ever assembled.

    A programme of such a size posed a problem: none of the companies involved wanted to carry tens of billions of dollars of AI chips on their balance sheets.

    [...]

    That challenge produced an unusual solution. Morgan Stanley helped arrange a private-credit vehicle, funded by outside investors, that buys the chips and leases them to Anthropic in an adaptation of the vendor-financing model Boeing and GE built to sell aircraft and engines.

    [...]

    Financing the chips solved only half of Google’s problem. The company also needed enough powered data centres to house them. “We have a schedule and we’re looking for capacity that will fit the schedule,” the Google executive said. “Crypto miners with excess capacity were helpful.”

    [...]

    People familiar with the matter said the Big Tech company had so far backstopped 10 developments with 2.4GW of power for TPUs. Google’s guarantees put it on the hook for as much as $44bn if all the leases go bad, though it marks the liability at $815mn on its balance sheet. It could also step into the leases itself.

    The Google team is now racing to put together additional data centre projects with enough power to ultimately house all of the 4.5GW of TPU hardware they’ve agreed to sell. “We’re spending a lot of time on [power] right now — all of our time,” said the Google executive.

    11 votes

  19. Comment on How US states are streamlining college admissions in ~society

    skybrian
    Link
    From the article: [...] [...] [...]

    From the article:

    More than 130,000 Georgia high school seniors will receive letters in October listing colleges and universities that are saving spots for them in their 2027 freshman classes—even though they never applied.

    Georgia is one of 19 states that have developed some form of a direct-admissions system. Using data about academic performance through 11th grade, colleges essentially apply to students—a reverse of the traditional application process that can create logistical and financial barriers to enrollment.

    “For a lot of high school seniors, maybe they aren’t sure if they are college material,” said Chris Green, the president of the Georgia Student Finance Commission, which helped launch the system, called Georgia Match, in 2023. “It’s a real eye-opening experience to say, ‘I’m already admitted to 45 schools.’”

    [...]

    Organizers from multiple sectors of state government believe Georgia’s is the largest and most comprehensive direct-admissions program in the country. The first such program began in Idaho in 2015 and other states have followed suit.

    [...]

    Organizers have found that eliminating friction from the admissions process can encourage students to give more consideration to in-state schools, helping states retain talent, said Melanie Heath, the strategy director for access at the Lumina Foundation, an organization that aims to expand postsecondary opportunities.

    And including a wide array of institutions, including technical schools, may help students who weren’t previously considering college to credential programs in high-needs fields, like skilled trades, she said.

    [...]

    In Georgia, students’ college options are hosted on the same state data platform they use to discuss career paths as early as middle school. Students can click “claim my spot” next to a college on their offer letter to finish the application process, and schools waive application fees in November every year to eliminate another hurdle.

    As it works to refine its program in bigger ways, the state has also made small tweaks to ensure students understand how it works, Green said. For example, organizers beefed up the envelope size for offer letters when they realized families had mistaken the business-sized envelopes they originally used as junk mail.

    7 votes