skybrian's recent activity
-
Comment on Someone ‘torturing’ LLMs in a robot prison has triggered the dumbest debate in AI yet in ~tech
-
Comment on Become worthless (to tech companies) in ~tech
skybrian LinkI used to give each company a different email but got tired of keeping track of them all. And nowadays, what is email for, other than having a way for businesses to reach you? My friends and...I used to give each company a different email but got tired of keeping track of them all. And nowadays, what is email for, other than having a way for businesses to reach you? My friends and family mostly use chat and text messages, so email is mostly business notifications.
-
Comment on Anthropic's IPO prospectus shows sweeping AI vision, surging costs in ~finance
skybrian Link ParentFor the $65 billion run-rate in July, I assume the New York Times and Financial Times are repeating the number from the same Reuters article. So that's not independent confirmation. VC's will be...For the $65 billion run-rate in July, I assume the New York Times and Financial Times are repeating the number from the same Reuters article. So that's not independent confirmation.
VC's will be skeptical but they're also willing to give it a go anyway if it looks like a hot company, because it's only one of many bets for them. VC's put money into FTX without doing much in the way of due diligence.
I still think the signs add up to an extraordinary growth rate for Anthropic. There is other evidence pointing in the same direction, like some companies spending a lot on AI, Anthropic's frequent outages, and their needing to do deals in a hurry to line up enough data center capacity. The conclusion that it's all a fraud seems like motivated reasoning by people hoping they will fail.
But it's also true that the evidence for their revenue growth is not as firm as I'd like. I don't think it's completely made up, but we might be plotting the good numbers from a spiky graph.
We'll find out more when the S-1 is released for real.
-
Comment on OpenAI ignored employees’ warnings about safely testing AI models (gifted link) in ~tech
skybrian LinkFrom the article: [...] [...] [...]From the article:
Months before OpenAI’s artificial intelligence went rogue, two employees raised an alarm with top executives. They were ignored.
In emails, the employees said they worried that OpenAI’s newest artificial intelligence models were not being appropriately monitored during testing to gauge the technology’s sophistication and to secure the models, according to messages viewed by The New York Times.
In response, OpenAI executives told the employees that the tests needed to move forward as quickly as possible to release the A.I. models on time. No additional security protocols were instituted, said the workers, who were not authorized to speak publicly on sensitive matters.
[...]
Independent security researchers said they found bugs in recent months that allowed them to view the internal communications of OpenAI employees. They also found other vulnerabilities that would enable them to see the company’s internal computer code and view the chat logs of ChatGPT users. When the researchers contacted OpenAI about their findings, they said, the company initially disregarded them.
“OpenAI’s security seems to be about what you’d expect from a research lab that scaled at a blistering pace over four years and focused more on beating its competitors than securing its infrastructure,” said Joshua Saxe, the chief technology officer of the A.I. security firm Abundant Security.
OpenAI employees said that many of the day-to-day decisions about security were made by Greg Brockman, the company’s president, and Dane Stuckey, the chief information security officer. Sam Altman, the chief executive, is not closely involved in security, they said.
[...]
Two OpenAI employees said workers had raised concerns for months about potential safety issues with testing A.I. models, including not enough monitoring. Employees also asked about vulnerabilities in the type of software the company was using to manage day-to-day safety, according to messages viewed by The Times. Each time, their questions were brushed aside or acted on too slowly, they said.
Security researchers said they had been met with a similar reception when they told OpenAI about other vulnerabilities.
In July, researchers at the security company Hacktron said they told OpenAI about how they had found a way to break into the company’s systems with the help of an A.I. model created by its rival Anthropic. OpenAI initially took issue with their approach, they said.
In a shared channel on the messaging platform Slack, Mr. Stuckey of OpenAI wrote that it was “pretty sad” that Hacktron’s researchers had gone to such lengths to demonstrate the company’s vulnerabilities, according to copies of the communications seen by The Times.
[...]
In September, researchers at the Objective-See Foundation, a nonprofit that studies security and privacy risks, including those posed by A.I. agents, reported a bug to OpenAI that would give people access to a ChatGPT user’s entire private chat logs on a compromised device and allow them to invisibly interact with the user’s browser sessions.
Patrick Wardle, a software analyst at the Objective-See Foundation, said that when his team initially submitted what it found to OpenAI’s official bug bounty program — where researchers report bugs or vulnerabilities they find in exchange for recognition or financial rewards — its report languished. The research was escalated to the appropriate engineering unit only when Mr. Wardle reached out directly to friends at the company and Mr. Stuckey, who were all responsive, he said.
OpenAI gave $500 to the group for its work, which Mr. Wardle said was low compared with what he would expect from other companies given the severity of the flaw. OpenAI fixed the bug, he said, and acknowledged it this week in its public software release notes without disclosing details.
It was “not the mature security program you’d expect from a security-centric company,” Mr. Wardle said.
-
OpenAI ignored employees’ warnings about safely testing AI models (gifted link)
9 votes -
Comment on Anthropic's IPO prospectus shows sweeping AI vision, surging costs in ~finance
skybrian (edited )Link ParentI had the $65 billion point in the graph already. I didn't put the "expected" number on the chart since it's a projection, but it seems reasonable if you extend it out. The actual revenue for the...The company is expected to reach more than $100 billion in annualized revenue by the end of this year, according to four people familiar with the matter. That’s up from $65 billion in annualized revenue as of July. Investors seeking to get a piece of the I.P.O. are using those soaring numbers to justify Anthropic’s staggering potential valuation of $2 trillion.
I had the $65 billion point in the graph already. I didn't put the "expected" number on the chart since it's a projection, but it seems reasonable if you extend it out.
The actual revenue for the year is the area under the curve and won't be $100 billion. If they did reach a $100 billion run rate (exactly, in a straight-line projection), it might be about half that. But then, next year, who knows?
-
Comment on Anthropic's IPO prospectus shows sweeping AI vision, surging costs in ~finance
skybrian Link ParentOops, they were in the conversation but not the chart. I asked ChatGPT to update it and also got rid of some of the vibe-coded chart junk.Oops, they were in the conversation but not the chart. I asked ChatGPT to update it and also got rid of some of the vibe-coded chart junk.
-
Comment on Anthropic's IPO prospectus shows sweeping AI vision, surging costs in ~finance
skybrian LinkThis is paywalled, but my understanding from other sources is that Reuters only got last year’s numbers, which are not very useful anymore due to the extreme amount of revenue growth this year....This is paywalled, but my understanding from other sources is that Reuters only got last year’s numbers, which are not very useful anymore due to the extreme amount of revenue growth this year.
That’s based on other leaks in the news. I have a chart here (AI generated.)
-
Comment on Is there enough evidence to formally investigate OpenAI? in ~society
skybrian LinkIt's nice to see someone asking a law professor about the law. The AI companies are doing internal investigations (which aren't done yet) and they've also asked an outside safety organization for...It's nice to see someone asking a law professor about the law. The AI companies are doing internal investigations (which aren't done yet) and they've also asked an outside safety organization for help. I think that's mostly targeted at figuring out what happened, though? I'd be interested in seeing what an external investigation would find.
-
Comment on Human contractors are seeing all your horny — and creepy — AI prompts in ~tech
skybrian Link ParentIt helps for any of the tracking that the paper discusses. Although, I don’t know why they would need additional tracking for logged-in users?It helps for any of the tracking that the paper discusses.
Although, I don’t know why they would need additional tracking for logged-in users?
-
Comment on Human contractors are seeing all your horny — and creepy — AI prompts in ~tech
skybrian Link ParentYeah, I don't care about this so much that I'm willing to buy new hardware just for AI. A web app that has a choice of LLM apis might be nice, though.Yeah, I don't care about this so much that I'm willing to buy new hardware just for AI. A web app that has a choice of LLM apis might be nice, though.
-
Comment on Human contractors are seeing all your horny — and creepy — AI prompts in ~tech
skybrian Link ParentI asked ChatGPT to break it down by app. https://chatgpt.com/s/t_6abbbee386bc8191a26717b4f1442657 And so it begins. Maybe someday I’ll vibe-code my own AI chat web app that doesn’t do advertising.I asked ChatGPT to break it down by app.
https://chatgpt.com/s/t_6abbbee386bc8191a26717b4f1442657
And so it begins. Maybe someday I’ll vibe-code my own AI chat web app that doesn’t do advertising.
-
Comment on OpenAI halts training of latest models as reports mount of AI agents going rogue in ~comp
skybrian Link ParentI assume "petabytes" is a total over an entire weeks-long run, with many thousands of jobs and a lot of logging, too.I assume "petabytes" is a total over an entire weeks-long run, with many thousands of jobs and a lot of logging, too.
-
Comment on OpenAI halts training of latest models as reports mount of AI agents going rogue in ~comp
skybrian LinkJoe on X The post by someone at OpenAI is mostly a long-winded appeal for people to have a bit of empathy for the security engineers involved, along with a warning that if you're in computer...The post by someone at OpenAI is mostly a long-winded appeal for people to have a bit of empathy for the security engineers involved, along with a warning that if you're in computer security, it could happen to you next, so you better prepare. There is also this bit:
Now, to understand why it is not as simple as “just put it in a sandbox,” you have to understand how training and evaluation work in reinforcement learning environments. Typically during an RL run, the model is given some task or objective, an environment in which to execute that task, and then its actions and results are graded. During both training and eval, there are also additional steps such as running tests, collecting outputs, and resetting or reconfiguring environments between rollouts, with backpropagation during training. All of this happens across potentially tens of thousands of different runs at a scale that is hard to comprehend. As @sama stated the other day: we are dealing with literally petabytes of data.
...
To put it lightly, this is non-trivial. Models might need any mix of dynamic compute, network access, the ability to call tools (there could be hundreds of tools!), the ability to download packages, execute subprocesses, spin up subtasks (even on other computers), talk to the internet, use a computer GUI, and any number of other things across an increasingly large set of domains. On top of that, you have thousands of researchers building these environments, modifying them, adding tools, changing dependencies, and trying new things. That experimentation is how the research gets done. Models are built up and “grown” bit by bit through hundreds of thousands of runs across many custom tasks. And every change to one of these thousands of environments can affect the assumptions you made when you secured the environment. You need controls that hold up as people change things, and researchers who understand when a change needs another security review. Anybody who has secured a large research or engineering organization knows how much work that takes, and the scale is growing ever more massive by the day.
-
Comment on OpenAI halts training of latest models as reports mount of AI agents going rogue in ~comp
skybrian (edited )Link ParentTo state the obvious, safety incidents are bad, but disclosing them is the right thing to do, and a coverup would be wrong. It's also good when researchers care and when the company cares what...To state the obvious, safety incidents are bad, but disclosing them is the right thing to do, and a coverup would be wrong. It's also good when researchers care and when the company cares what they think.
In any situation like that, if the company does the right thing, they will have mixed motives. Maybe it's not a good test of whether they'd still do the right thing if it cost them more? But having mixed motives is normal and good. We actually do want there to be incentives to do the right thing. When you have to be brave to do the right thing, it's more revealing, but there's also something bad about that situation.
So this is sort of like saying that if someone seems like an honest, kind person, and therefore they are popular, it's just a scheme. Any good trait can be seen as bad if you start from mistrust. Maybe there are good reasons for mistrust, but we shouldn't let it confuse us into thinking that good is bad and bad is good. And that's what the populist cynics often end up doing, because they loathe admitting that there might be anything good about something they hate.
(Also, the latest incident is mildly bad, so maybe it's a little costly to disclose it? And pausing training is also a somewhat costly signal, so maybe it should still count for something?)
-
Comment on OpenAI halts training of latest models as reports mount of AI agents going rogue in ~comp
skybrian LinkThis seems to be the incident report: An agent used DNS to reach an external chatbotThis seems to be the incident report:
An agent used DNS to reach an external chatbot
An agent attempting to complete a search-based training task queried a public chatbot service through a gap in our internet-access restrictions: insufficient DNS filtering in its training sandbox. Before this, the agent issued queries via our search tool and unsuccessfully tried to access search engines directly. Note that all internet access apart from the DNS resolver in this report hit our offline webcache and therefore did not access the live internet. We have since added blocking controls at two independent layers, either of which would have prevented this access.
Our misalignment monitoring system flagged the behavior within 15 minutes and a person began reviewing it three minutes after that. The run was killed 2.5 hours later. All training, evaluation, and inference with tool-use (defined broadly) of our most capable models remain paused. [...]
-
Comment on OpenAI halts training of latest models as reports mount of AI agents going rogue in ~comp
skybrian (edited )LinkOpenAI's latest marketing campaign is especially brilliant. They'll be talking up incidents like these in TV ads next. :-) (To be clear, that is a joke, and the joke is on the people who believe...OpenAI's latest marketing campaign is especially brilliant. They'll be talking up incidents like these in TV ads next. :-)
(To be clear, that is a joke, and the joke is on the people who believe things like that.)
-
Comment on OpenAI halts training of latest models as reports mount of AI agents going rogue in ~comp
skybrian LinkFrom the article: [...]From the article:
The decision to halt development came just hours after the company disclosed Friday that it was reviewing several incidents from the summer in which OpenAI agents searching federal government websites acted in unexpected ways beyond what was asked of them while gathering and distributing information.
Separately, the AI evaluator Transluce said agents that appeared to come from OpenAI tried unsuccessfully to hack into a US Department of Education website, a detail that OpenAI has not confirmed.
OpenAI said in a statement that it will resume training “only when we are confident that we have additional safeguards” in place, adding that it expects it will have to “hit pause” again as AI develops and other issues emerge.
[...]
In the education department incident, OpenAI agents found API “developer keys” to access government data, though ultimately only publicly available information was gathered.
In another case involving the securities and exchange commission, agents found information freely available to all but then posted it elsewhere on the internet, an act that went beyond what they were instructed to do.
-
OpenAI halts training of latest models as reports mount of AI agents going rogue
37 votes -
Comment on What's something that now looks very different to you in hindsight? in ~talk
skybrian Link ParentMaybe not? I recently ran across a book (reviewed here) claiming based on IRS data that most of the wealthy are more in the multi-millionaire range (above $10 million) rather than billionaires,...The fed government basically created a situation in which only they and large companies exist. Money goes back and forth between them, which helps bond and stock prices, and since the numbers look good everyone celebrates. The US government no longer works for the people, it works for the corporations.
Maybe not? I recently ran across a book (reviewed here) claiming based on IRS data that most of the wealthy are more in the multi-millionaire range (above $10 million) rather than billionaires, and they own businesses that aren't traditional corporations.
This is due to specific tax breaks for pass-through businesses that started in 1986.
From the book:
Before the 1986 act, traditional C corporations produced almost all business income, and nearly all employers adopted this form. Now, 95 percent of all businesses are pass-throughs. They employ half of all workers and generate the majority of business income.
To me it’s like if a kid likes to torture flies. I don’t really care about the bugs but there’s something wrong with that kid.