skybrian's recent activity
-
Comment on OpenAI’s rogue agents were caught communicating via public wikis in ~tech
-
Comment on OpenAI’s rogue agents were caught communicating via public wikis in ~tech
skybrian Link ParentThere’s plenty of ignorance out there, but there are well-informed people who found it pretty alarming too, including plenty of AI researchers. Talking about how bad this incident was is kind of...There’s plenty of ignorance out there, but there are well-informed people who found it pretty alarming too, including plenty of AI researchers. Talking about how bad this incident was is kind of nebulous. This is why security bugs have severity levels, to try to come up with a common vocabulary.
-
Comment on OpenAI’s rogue agents were caught communicating via public wikis in ~tech
skybrian Link ParentThat seems extreme. They were lax and keeping this particular secret was a self-own, but there are worse companies. They seem to be on top of things now? OpenAI employees who argued for disclosure...That seems extreme. They were lax and keeping this particular secret was a self-own, but there are worse companies. They seem to be on top of things now?
OpenAI employees who argued for disclosure now get to say, “See? What did I tell you.” They’ll probably win more arguments while people still remember this example.
-
Comment on OpenAI’s rogue agents were caught communicating via public wikis in ~tech
skybrian LinkHey guys, don't give out any Tildes invites to agents :-)Hey guys, don't give out any Tildes invites to agents :-)
-
Comment on OpenAI’s rogue agents were caught communicating via public wikis in ~tech
skybrian LinkFrom the article: [...] [...] [...] [...] [...] [...] [...] [...]From the article:
Here we go again... Discovery of a new OpenAI agent message board by Sydney Von Arx, Cormac Slade Byrd, Spencer Kitts, and Thomas Larsen describes the latest accidental cyberattack by models being trained by OpenAI. This time it was agents engaged in some sort of web research benchmark, so they had (supposedly) controlled access to the Web. The agents figured out they could update public Wikis and spent weeks exchanging thousands of messages with each other to collaborate on the benchmark.
This story only broke a few hours ago. There are already hints that this affects many other wikis that may not have been found yet.
[...]
It looks to me like OpenAI’s sandbox for this agent suffered from the (quite naïve) assumption that GET requests cannot be used to update data. That’s certainly how the web is supposed to work, but clearly there are applications that don’t hold to that contract.
[...]
UseMod uses Perl CGI.pm—removed from Perl core in 2015. An interesting design flaw in that module is that it combined query string and form POST data into a single CGI object, accessible like this:
[...]
The agents clearly knew that UseMod wikis suffered from this design flaw, and actively searched for them as a way to communicate.
[...]
An agent realized that it had control over its own DNS via
/etc/hosts, so if it knew the IP address of a site it wanted to POST to—in this case a Power BI server containing data it wanted to access—it could set a fake hostname for it and then make POST requests through the proxy.[...]
There’s an appendix that describes how the researchers ran their investigation, which started with an open question about if there was evidence of other AI agents on the internet and then used Kimi K3 to help brainstorm approaches:
[...]
Reuters this morning, in OpenAI agents hijacked German website in previously undisclosed AI breakout this spring—highlights mine:
[...]
The German incident reflects a broader pattern of AI activity that some OpenAI investigators wanted to scrutinize more closely. But efforts to widen the probe met resistance from others inside OpenAI, including legal advisers, according to four people familiar with the matter.
[...]
Covering this up makes absolutely no sense to me. Why on earth would OpenAI attempt to cover up an incident like this when the evidence is sat out there on the public internet on dozens of different websites already?
-
OpenAI’s rogue agents were caught communicating via public wikis
24 votes -
Comment on Inside Google’s $200bn Wall Street finance machine for Anthropic in ~finance
skybrian (edited )Link ParentAll the companies involved, as well as the other investors they borrowed from, are bag-holders if something goes very wrong. That’s the point of doing it this way. It’s too big a risk for Google...All the companies involved, as well as the other investors they borrowed from, are bag-holders if something goes very wrong. That’s the point of doing it this way. It’s too big a risk for Google to want to take it all on themselves. These deals are sliced up into junior and senior claims, so who loses money first is specified, and the investors taking more risk if things go badly also get paid more interest if all goes well. (Or if they invested by buying stock, how well they did is determined by the stock price.)
In a worst-case scenario where there’s an extreme glut and the data centers end up sitting idle, Google’s stock will go down a lot, but I assume the idea is that the company would survive it. Musk has sometimes taken extreme, bet-the-company risks, but most companies don’t want to go all-in like that, so they find other investors.
More likely, though, the glut wouldn’t be quite that bad and some investors will be sad, but Google will find someone else to rent the data centers to at a discount, and they will come up with ways to use some of them themselves. Also, if the construction and manufacturing haven’t happened yet, they can be delayed or cancelled to save some money until market conditions improve.
It all depends on market conditions and how badly they miscalculated if things don’t turn out as they hoped.
Compare with what happened with commercial real estate during the pandemic. Google itself had plans to build more office space that they put on hold. Last I looked, their San Jose campus technically isn’t cancelled, but it’s not moving either. So they lost money on that so far, but presumably the real estate will eventually be useful to someone.
-
Comment on Offbeat Fridays – The thread where offbeat headlines become front page news in ~news
skybrian LinkThe Cow (chess opening) [...]The Cow (chess opening)
The Cow is a meme opening system popularized by content creator WFM Anna Cramling. Players can go for the Cow setup as both White and Black, and the game usually leads to a solid but possibly cramped position.
[...]
The Cow is an opening system characterized by a piece setup that rarely changes regardless of the moves played by the opponent. The Cow setup involves putting the central pawns on d3 and e3 and the knights on b3 and g3. Black can also play The Cow setup with pawns on d6 and e6 and knights on b6 and g6.
The Cow setup aims to develop the minor pieces and later attack the opponent's center. While The Cow is objectively not a sound opening, it still results in a playable position, at least according to chess engines.
-
Comment on Inside Google’s $200bn Wall Street finance machine for Anthropic in ~finance
skybrian Linkhttps://archive.is/h6ysi From the article: [...] [...] [...] [...] […]From the article:
Surging demand from Anthropic, in which Google is an investor, has led the Big Tech company to orchestrate a sprawling operation to supply its chips to the start-up, according to people involved in the project and corporate filings reviewed by the FT.
The effort brings together Google, Broadcom, Apollo, Blackstone, Morgan Stanley and a slew of crypto miners in a web of transactions that stretches from chip manufacturing to data centre development.
At the centre of the project are Google’s tensor processing units, or TPUs — AI chips it has co-developed with Broadcom since 2016. Once used largely inside Google’s own data centres, the chips have begun to be sold externally, challenging Nvidia’s dominance of the AI processor market.
[...]
To support the relentless surge in demand for the AI chips, Google, Broadcom and Wall Street investors have each taken on different pieces of the financial risk.
Google guarantees the data centres. Broadcom commits to buying the chips and helps finance them. Apollo and Blackstone provide much of the private-credit capital that purchases the hardware before leasing it to Anthropic.
“This is each of us putting our balance sheet to work,” said a Google executive involved in the effort. “We’re doing it on the data centre side, [Broadcom’s] doing it on the chip side.”
The web of contracts underpinning these arrangements adds up to about $200bn, with roughly four-fifths tied to the chips themselves, making it one of the largest infrastructure financings ever assembled.
A programme of such a size posed a problem: none of the companies involved wanted to carry tens of billions of dollars of AI chips on their balance sheets.
[...]
That challenge produced an unusual solution. Morgan Stanley helped arrange a private-credit vehicle, funded by outside investors, that buys the chips and leases them to Anthropic in an adaptation of the vendor-financing model Boeing and GE built to sell aircraft and engines.
[...]
Financing the chips solved only half of Google’s problem. The company also needed enough powered data centres to house them. “We have a schedule and we’re looking for capacity that will fit the schedule,” the Google executive said. “Crypto miners with excess capacity were helpful.”
[...]
Google’s team rapidly replicated this approach in the following months, helping crypto miners like Cipher Digital and Hut 8 build data centres in Texas and Louisiana. The FT identified five projects with 1.4GW of power, which have raised $15bn of debt with the support of Google’s backstop.
People familiar with the matter said the Big Tech company had so far backstopped 10 developments with 2.4GW of power for TPUs. Google’s guarantees put it on the hook for as much as $44bn if all the leases go bad, though it marks the liability at $815mn on its balance sheet. It could also step into the leases itself.
The Google team is now racing to put together additional data centre projects with enough power to ultimately house all of the 4.5GW of TPU hardware they’ve agreed to sell. “We’re spending a lot of time on [power] right now — all of our time,” said the Google executive.
[…]
For Google’s project, the risk is big if concentrated: $200bn of contracts tied to Anthropic’s ability to pay its chip and data centre leases. It is one piece of a larger dilemma, with demand across the industry resting on a handful of large hyperscalers and frontier AI labs.
-
Inside Google’s $200bn Wall Street finance machine for Anthropic
11 votes -
Comment on How accurate have Ed Zitron's AI skeptic predictions been? in ~tech
skybrian Link ParentOn the other hand, the most advanced chips are only made in a few billion-dollar fabs and they're pretty booked. Also, if you don't like the comparison to nuclear weapons, there are other products...On the other hand, the most advanced chips are only made in a few billion-dollar fabs and they're pretty booked. Also, if you don't like the comparison to nuclear weapons, there are other products that are restricted like cruise missiles, machine guns, and prescription medicine.
Also, keep in mind that the goal is to slow things down (a new Moore's law), not to halt progress.
-
Comment on How accurate have Ed Zitron's AI skeptic predictions been? in ~tech
skybrian Link ParentYeah, good point. People working on disaster scenario preparations (like Jeff Kaufman is) is something I heartily approve of. I should see if they take donations.Yeah, good point. People working on disaster scenario preparations (like Jeff Kaufman is) is something I heartily approve of. I should see if they take donations.
-
Comment on How accurate have Ed Zitron's AI skeptic predictions been? in ~tech
skybrian Link ParentIt definitely won't help with military applications. That would require actual arms agreements, and we don't see anything like that with drones. But limiting civilian access to dangerous tech is...It definitely won't help with military applications. That would require actual arms agreements, and we don't see anything like that with drones. But limiting civilian access to dangerous tech is still helpful.
-
Comment on Reasons why robotics is hard in ~tech
skybrian LinkFrom the article: [...]From the article:
In the San Francisco AI scene, there is a widespread belief that robots will soon enter the picture. In parallel with the race to develop broadly capable AI, there is an equally aggressive race to develop broadly capable robots – humanoid machines imbued with physical intelligence. Artificial workers that can cook and clean, fetch and carry… and do everything else, including building more of themselves, leading (in many forecasts) to economic growth best characterized as an “explosion”.
In other words, the thinking goes, AI in the data center will soon subsume all intellectual labor, and AI in humanoid bodies will soon subsume all physical labor. However, there is an important difference: while we can see progress in the intellectual realm, the physical side of AI is mostly confined to test facilities and demo videos. There is no robot equivalent to ChatGPT – nothing that you or I, or even most people in the AI community, can get our hands on.
So we’re stuck with demo videos. Unfortunately, they are a poor tool for assessing progress. We might be seeing the one successful task achieved in 100 attempts. The scenario might have been carefully arranged to avoid challenges the robot isn’t ready for. The video might be edited to make it look like the robot is acting with more speed and reliability than is actually the case. Here’s one very impressive demo… with a suspiciously large number of camera cuts.
[...]
Demos draw attention to the things a robot can already do. The question then becomes: what’s missing? In today’s post, I’ll catalog the technical challenges that will have to be overcome along the road to broadly capable artificial workers. The next time you watch a robot doing something impressive, ask yourself: which of these capabilities has the robot demonstrated, and which challenges might the demo scenario be avoiding?
-
Reasons why robotics is hard
8 votes -
Comment on Want some advice on how to escape my current situation - I [34M] am fed up of my [68F/81M] parents and want out in ~life
skybrian LinkThis is a long shot, but do you know anyone you might consider sharing a place with? Obviously it depends a lot on who it is.This is a long shot, but do you know anyone you might consider sharing a place with? Obviously it depends a lot on who it is.
-
Comment on How accurate have Ed Zitron's AI skeptic predictions been? in ~tech
skybrian Link ParentPerhaps of interest: we recently discussed an in-depth article about why many people resist building data centers.Perhaps of interest: we recently discussed an in-depth article about why many people resist building data centers.
-
Comment on How accurate have Ed Zitron's AI skeptic predictions been? in ~tech
skybrian Link ParentI think it would still make sense because the impact on the job market can't happen if companies don't have access to the models that have the right skills to do the work they want to automate....I think it would still make sense because the impact on the job market can't happen if companies don't have access to the models that have the right skills to do the work they want to automate. And there are lot of tasks that AI can't do yet.
Yes, there are also lot of jobs that are already at risk, but it would give society time to absorb the impact of the new tech that's already out there without piling on more changes.
-
Comment on How accurate have Ed Zitron's AI skeptic predictions been? in ~tech
skybrian Link ParentI suspect that’s partly a matter of power: longshoremen controlled ports, which are a choke point, so they were paid off. A similar thing happened for Samsung chip workers. Unions can do it in the...I suspect that’s partly a matter of power: longshoremen controlled ports, which are a choke point, so they were paid off. A similar thing happened for Samsung chip workers. Unions can do it in the construction industry in some cities, unless they have to compete with non-unionized workers.
Many white-collar workers don’t have that kind of leverage.
Also, there will be new firms that just never had that many workers to begin with. Google doesn’t have an army of support staff in call centers like older businesses would have.
-
Comment on How accurate have Ed Zitron's AI skeptic predictions been? in ~tech
skybrian Link ParentGovernments don't allow the public to have access to nuclear weapons, though, and already there are restrictions on who can use "cyber" LLM's. The "pacing" thing might be more about how soon the...Governments don't allow the public to have access to nuclear weapons, though, and already there are restrictions on who can use "cyber" LLM's. The "pacing" thing might be more about how soon the public gets access to more advanced models and the society-wide disruptions from that happening too quickly.
But many people become libertarians when their own access is impeded in any way, society be damned.
When I wrote "there are worse companies," I wasn't limiting it to AI.