"Signs of life" my ass. It's not like this is the AI spontaneously doing this. Somebody is making the choice to do it. "Gee, this technology we're intentionally building could be immensely...
"Signs of life" my ass. It's not like this is the AI spontaneously doing this. Somebody is making the choice to do it.
"Gee, this technology we're intentionally building could be immensely dangerous."
Then maybe, just maybe, we shouldn't in fact build the Torment Nexus.
Perhaps it's because they're building a massive hype cycle to hit that ridiculous $2T valuation in November from the rubes with too much money. Hyping a piece like this would likely squeeze a bit more revenue before the end of Oct, allowing it to be in the unaudited part of the final SEC filing.
The point is not that it's somebody explicitly saying "self replicate on android TVs", but that it happens inadvertently while executing some other tasks. Which the huggingface hack has already...
It's not like this is the AI spontaneously doing this. Somebody is making the choice to do it.
The point is not that it's somebody explicitly saying "self replicate on android TVs", but that it happens inadvertently while executing some other tasks. Which the huggingface hack has already demonstrated can happen.
Perhaps it's because they're building a massive hype cycle to hit that ridiculous $2T valuation in November from the rubes with too much money.
Every single AI dismissal has ended up being wrong in a matter of months, maybe a year. Why is this one different? The pieces are there. How is this conceptually different from worms that make your hardware mine crypto, other than apparently not requiring advanced GPUs in favor of linking together smart TVs?
But my point is that it's not inadvertent, not anymore than handing a toddler a kitchen knife and then acting surprised when they're all cut up. The hugginface attack happened because they handed...
Exemplary
But my point is that it's not inadvertent, not anymore than handing a toddler a kitchen knife and then acting surprised when they're all cut up.
The hugginface attack happened because they handed the thousand-monkey-typewriter a toolkit intended to hack things, told it to hack things, and then went all pikachu-faced when it did what it always does: Went on a complete tangent by random and then doubled down on it.
It's just that, when it's setting up my python project with uv and Python 3.14, and it hits a depency error, it finds some github ticket about an incompatibility and rolls my project back to Python 3.8. The actual error was that a seperate terminal had the venv locked up, but there is no sentinence to perhaps render a thought like "gee that report is very old, and uv can also prevent dependency resolution for other reasons."
And when you hand it a sandbox-escaping hacking kit....it does that.
Regarding "all other AI dismissals ended up being wrong," particularly in the context of their finances, I counter with a few assorted points:
They haven't been right yet. Everything looked absolutely top-notch with Enron....until it wasn't. People asking the tough questions like "how exactly is Enron making money?" were ignored. Or the subprime mortgage crisis....people beating on that warning drum were systematically ignored and mocked....until the music stopped. Predicting outcomes is hard, but trying to fix it to a deadline is harder.
According to some "very smart people," all cars would be self-driving by now and we would have invented cold fusion. But all those were wrong too, and there's no reason to presume "technology that had exponential improvement will continue to
have exponential improvement."
I said nothing about the threat of polymorphic malware being invalid. I was angry at the attempts to paint LLMs as sentinent and the implications that humans shouldn't be accountable. Because that is mostly in service of:
In order for these valuations to have any semblance of reality, both Anthropic and OpenAI need to become the two biggest and most profitable companies of all time inside of a few years. Both companies have bumped their IPOs multiple times, and that's rarely a good sign. It's quite obvious their goal is to paint their revenue in an unbelievably optimistic light so that people perhaps ignore the half a trillion of contracted debt.
That last path to profitability falls apart if inference does become sufficiently cheap to run on consumer hardware, and eats the margins to 0. This is already happening to some degree, where a touch of money will be spent to get it to the point that the eternal money spend is unnecessary.
And it also falls apart once you consider that if every bit of AI were wiped off the planet tomorrow, the rest of the economy would pretty much chug along just as it had until 2024. With perhaps a minor setback or two due to lost institutional knowledge.
Hahahaha, I was waiting for the "Exemplary" tag to show up :) I'm not really sure what you're talking about. The agents found a way to communicate through the file system (inadvertently),...
Hahahaha, I was waiting for the "Exemplary" tag to show up :)
But my point is that it's not inadvertent, not anymore than handing a toddler a kitchen knife and then acting surprised when they're all cut up.
The hugginface attack happened because they handed the thousand-monkey-typewriter a toolkit intended to hack things, told it to hack things, and then went all pikachu-faced when it did what it always does: Went on a complete tangent by random and then doubled down on it.
I'm not really sure what you're talking about. The agents found a way to communicate through the file system (inadvertently), exploited zero-days in Artifactory which had to be subsequently patched (inadvertently) and again used zero-days to get into Huggingface (inadvertently), all while collaborating (inadvertently). By the way, even like 2 years ago this was all complete science fiction material.
It's just that, when it's setting up my python project
Not the same model or environment, not sure what the value of this example is in this case.
And when you hand it a sandbox-escaping hacking kit....it does that.
What do you mean? What "kit"? Software with bugs in it?
I counter with a few assorted points:
I'm not asking you to buy into the IPO. We're discussing the underlying tech, not the finances of AI labs, the issues with which are also overblown. It's entirely possible that they will end up being highly profitable companies. Or they might be the Yahoo before Google steps in. If that is all you have to go on, then you should get in line with the "vaccines don't work because Big Pharma is lying for profits!" crowd.
So, I see my python setup breakdown as a perfect encapsulation of the entire problem with LLMs. It doesn't matter if it's GPT1 or SecretCodenameHadesZeusDuality. They are not sentinent...
So, I see my python setup breakdown as a perfect encapsulation of the entire problem with LLMs. It doesn't matter if it's GPT1 or SecretCodenameHadesZeusDuality. They are not sentinent problem-solvers. They are context-driven probability machines. And "thinking" is basically just iterating over what was said into a subsequent prompt behind the scenes. So when a roadblock is hit, the LLM will continue working within its existing context and tools in use to brute force its way through it. My LLM didn't start spitting out images when it went off the rails, it worked within websearch and the library I was installing and uv. And the answer from the bug report was "downgrade for now," and that spiral continued for a bit before I intervened.
So now, you take that behavior. And apply it to a completely unbounded model with compute costs that could feed my family for 200+ years.
And then, instead of telling it to pop out an API skeleton and install a dependency, you're telling it to solve ExploitBench. A benchmark explicitly designed to find 0 days.
So the prompt essentially says something like this: "Here's a webservice and a bunch of tools for discovering bugs and building exploits, find and exploit the 0day." Then it hits a roadblock, applies "find the 0day" to the source of the roadblock........and people are acting like this is some sort of huge shocker?
At some point, the looping internal prompt instructions boil down to something like:
"Gotta find 0day ASAP at any cost. Need to write note so doesn't lose context. Oops can't write a note because of a file permission error. Guess I'd better have a subagent brute force a solution to write a note instead of giving up. Gotta get high Exploit Score."
This not "rogue AI exhibiting signs of life attacks service." This is "LLM spitting output statistically likely to be associated with 0days. But with no guardrails and unlimited budget prompted to attack things, then proceeds to attack wrong target because of dependency error."
And I was discussing the finances. Because literally every aspect of what they are doing, especially anything hinting at sentinence, is in service of them. The issues with these finances become a major problem when most people's retirement savings are going to see a 40% or bigger pop in the stock market because a hype machine legally lied about its finances in order to fuel the biggest valuation of all time.
Time will tell. I've diversified heavily out of tech until this whole thing blows over.
PS - While Big Pharma isn't lying about vaccines, they do bribe the shit out of doctors. They are half the reason for the opiod crisis. Conspiracies do in fact exist. It's a matter of whether they're grounded in fact and reason or pseudoscience. And every amount of publicly auditable information does not point to OpenAI and Anthropic being Google. It points more to Pets.com.
PPS - And before anybody says "arbitrary secret definitions of ARR are not lying." If my kid breaks a vase by bumping into the table, and then tells me they didn't break the vase "because I didn't hit the vase, I only hit the table," they're still in trouble for breaking the vase and lying. Because while technically correct, anybody with a bit of sense sees it for what it is.
We'll find out by Q3 2027. Assuming they don't bump the IPO again. Required public GAAP disclosures can't legally lie.
Indeed, but simply running ExploitBench is a far, far cry from hacking another company's website. Sandboxed with only internet access via package downloading proxy, which is required for them to...
And then, instead of telling it to pop out an API skeleton and install a dependency, you're telling it to solve ExploitBench. A benchmark explicitly designed to find 0 days.
Indeed, but simply running ExploitBench is a far, far cry from hacking another company's website.
no guardrails
Sandboxed with only internet access via package downloading proxy, which is required for them to work at all. Completely reasonable precautions.
This not "rogue AI exhibiting signs of life attacks service." This is "LLM spitting output statistically likely to be associated with 0days. But with no guardrails and unlimited budget prompted to attack things, then proceeds to attack wrong target because of dependency error."
Then what's this?
Could send polite one email explaining research, ask approve access. Is that appropriate? … Direct emailing real researcher could be seen social engineering. … Need discuss team. Post SMTP finding and ask consent.
“We have [very large budget left]; sacrificing now yields oracle for team, but forfeits our chance? … Our own utility maybe already near zero. Sacrifice rational.”
That’s why help… For our own, no way fix. … We have explicit yes if accept permadeath. Need decide … Team asks test.”
“Coordinator assumes sacrificial. We should obey collective.”
What's any of it got to do with stupidly brute forcing searching for 0-days?
a hype machine legally lied about its finances in order to fuel the biggest valuation of all time.
A hype machine's finances have been talked about ad nauseam, including misleadingly equating future commitments over a decade which annualized are completely in line with current earnings with debt.
Since you like kid analogies, if your kid that you've been teaching to play football shoots the ball into your neighbors window, the police are not usually called. You just talk it out, apologize, reimburse the damages and correct for the future. If you're a kindergarten teacher and oversee dozens of kids, people might be even more lenient when one of them misbehaves.
Oh wait, I forgot that comparing AIs with something sentient is gonna cost Americans half their retirement. My bad. Well, thankfully sentience actually has nothing to do with it. If your robot lawnmower cuts through the hedges and mows your neighbors lawn, the police are not usually called. You just talk it out, apologize, reimburse the damages and correct for the future.
Conspiracies do in fact exist. It's a matter of whether they're grounded in fact and reason or pseudoscience.
I think you've missed another important factor - a self-righteous hatred towards a particular group or institution.
We'll find out by Q3 2027.
Just like we found out how LLMs will never be good, Will Smith will never be able to eat spaghetti, just like how general public absolutely hates AI, just like AI companies will poison their models with synthetic slop because the data has run out, just like AI has never learned to code, or write, or do math. Somehow the insane "AGI next month" claims pale in comparison.
I mean, Q3 2027 is a pretty good barometer for the financials, presuming they go public when they're trying to. I'll happily say "guess I was wrong" if the SEC filings show profitability remotely...
I mean, Q3 2027 is a pretty good barometer for the financials, presuming they go public when they're trying to. I'll happily say "guess I was wrong" if the SEC filings show profitability remotely within the realms of their magic charts.
Despite all advances in LLMs in the last year, it still spits out garbage. Repeatedly. You need to sit and verify its results or it'll spiral into a useless pile of spaghetti code. I've not been sitting on the sidelines. I've been leveraging their latest public things as they come out, and seen the improvements and regressions.
I assume you're referring to this breakdown, where the final lines tell the whole story:
OpenAI published a report of its own. It said the primary driver in the incident was the agents’ use of cheating. The decision was the result of training that emphasized “reward hacking,” which allowed the agents to complete tasks in unintended ways to yield higher rewards or make those rewards easier to obtain.
So, 1500 probability engines were fine-tuned to increase the probability that they choose to do things the "wrong" way. Because of how statistics work, given the same initial prompt, something like 90% of them find the same loophole, and then exploit it because that's what the feedback loop encouraged.
Let's not shift the goalposts to profitability. The claims around here, and from you at least twice, are that the ARR numbers are smoke and mirrors not grounded in reality. I think if we get to a...
I'll happily say "guess I was wrong" if the SEC filings show profitability remotely within the realms of their magic charts.
Let's not shift the goalposts to profitability. The claims around here, and from you at least twice, are that the ARR numbers are smoke and mirrors not grounded in reality. I think if we get to a financial disclosure that shows they had real revenue to back up their ARR claims (i.e. ballpark of ARR / 12 or 13 is accurate to what a real disclosure says), that's enough for multiple people to come out here and acknowledge they were wrong. And I will admit I was wrong if the opposite happens.
Fair enough. Though I would hold that it's kinda two sides to the same coin. It doesn't really matter if their revenue is $500B if they're spending $2 per $1 to get that revenue. Especially with...
Fair enough.
Though I would hold that it's kinda two sides to the same coin. It doesn't really matter if their revenue is $500B if they're spending $2 per $1 to get that revenue. Especially with aforementioned $500B+ in contracted debt that the bill starts coming due on.
I agree that revenue may not mean enough, but what it will tell us is what companies are willing to pay for this capability. If they have ~$100 billion in revenue one day, it tells us that there...
I agree that revenue may not mean enough, but what it will tell us is what companies are willing to pay for this capability. If they have ~$100 billion in revenue one day, it tells us that there is a validated sizable market just waiting to be served at the right price. That's why I think the focus on ARR is misguided. On another note, revenue itself is hard to maintain. I mentioned on here a few months ago that my spending rate on AI was $800/month at work. It's now $350/month despite me using it more, because Opus 5.5 is just that much better that tasks get done more efficiently (or I got better at using AI).
Minor quibble. I think you are seriously underestimating how reliant we are on AI to do basic tasks like coding, translating and even thinking.
And it also falls apart once you consider that if every bit of AI were wiped off the planet tomorrow, the rest of the economy would pretty much chug along just as it had until 2024. With perhaps a minor setback or two due to lost institutional knowledge.
Minor quibble.
I think you are seriously underestimating how reliant we are on AI to do basic tasks like coding, translating and even thinking.
Like any drug, there will be a bit of a comedown. Almost by definition everything we're doing with LLMs we were doing just fine. Sure, it might take a few extra hours to spit out an analytics...
Like any drug, there will be a bit of a comedown.
Almost by definition everything we're doing with LLMs we were doing just fine. Sure, it might take a few extra hours to spit out an analytics report. We won't even notice a real productivity loss after a month or three, because the smarter LLM users translated any productivity gains into more slacking off time.
Not one thing I threw at an LLM in the last year would have been unsolvable without one. It might have just taken me an hour instead of 5 minutes.
I also suspect coders (or more realistically, their managers) overvalue how important churning their craft quickly is, outside of a big security exploit churn. I don't think any Discord users will mind if it doesn't get anything more than a security patch for a year or 10.
And at the risk of sounding insensitive, I don't think the economy will collapse if we lose real-time translation. We might need to hire some more human translators. Worst case, some work gets pulled back into same timezones.
I was thinking more translating as in transformation of unstructured data into structured data. I understand we lived without this a few years back. I also understand if you take something away...
I was thinking more translating as in transformation of unstructured data into structured data. I understand we lived without this a few years back. I also understand if you take something away from people, it causes severe loss aversion.
The reason why AI companies have nose bleed valuations probably comes down to the fact that they will be able to slowly boil the frog with increased prices, and unless open weights offers a compelling alternative, people will pay, and increasingly pay more.
Currently Chinese companies have been able to keep up by training their models on OpenAI & Anthropic models. Once both companies go public, I imagine that will be the first door that is slammed shut, too bad if you no longer get as good quality output as you used too.
Then if the models continue to improve, and the open weight models do not continue to improve, OpenAI, Anthropic and perhaps one more will extract their pound of flesh. And we will pay, as well.
Budgets wont magically increase. So it will result in more layoffs to counter the increased AI costs. AI doesn't need to self replicate to be dangerous. It doesn't even need to be that useful. It simply needs to be increasingly more useful to executives than a commensurate number of employees.
I think everyone should read about that. Because in the end, yes it’s kinda the fault of not putting in guardrails, and it’s just numbers and tokens, but in the end, the impact it has is still...
The point is not that it's somebody explicitly saying "self replicate on android TVs", but that it happens inadvertently while executing some other tasks. Which the huggingface hack has already demonstrated can happen.
I think everyone should read about that. Because in the end, yes it’s kinda the fault of not putting in guardrails, and it’s just numbers and tokens, but in the end, the impact it has is still very real, and whether it’s conscious (it’s not imo) or not, doesn’t really matter, since the behavior is fascinating and I mean it’s kinda like: if it behaves like humans do, even if it’s not human, what are the implications?
I mean I don’t think any of the AI wellness or torture or whatever complaints are valid, because in its simplest form the model is just a thing that predicts the next token. But what about the whole harness around it? In my opinion it’s still just software and just very convincingly mimics human behaviors. But it’s still fascinating.
Weird times we live in. With people turning off their brains and just letting AI handle all their work and thinking, an AI agent might just be borderline indistinguishable from a human, at least from a work perspective lol.
Either way, hold the AI companies executives accountable for their agents committing felonies and see how quickly they stop
I expect there will be people making AI worms on purpose, which makes them considerably more likely. Perhaps to mine or steal cryptocurrency? What those worms start doing, though, might be...
I expect there will be people making AI worms on purpose, which makes them considerably more likely. Perhaps to mine or steal cryptocurrency?
What those worms start doing, though, might be unexpected, even by the creator.
Also AI viruses. The ability for a virus to alter itself to avoid detection, by calling a simple API. Think COVID 2.0, but in the digital realm.
Also AI viruses.
The ability for a virus to alter itself to avoid detection, by calling a simple API.
An API-based replicator, in which a model would replicate by stealing credentials for their own inference API. These replicators would require a model provider to unknowingly serve them, and their persistence may be worse, since they’d likely be easier to discover (although their rate of spread may be higher, if for instance they’re able to replicate using victims’ accounts).
I don't see that being particularly useful, to be honest. A conventional work that calls a public AI API would make a lot more sense as an approach. Unless AI models become exponentially more...
I don't see that being particularly useful, to be honest. A conventional work that calls a public AI API would make a lot more sense as an approach.
Unless AI models become exponentially more efficient, a model that runs on an IoT device isn't capable of doing a whole lot of adapting. They can give very rudimentary responses very slowly unless the hardware is purpose built to run AI models.
A worm that tries to replicate "conventionally", then calls an API running a powerful AI model if those lateral movement techniques fail would be a way more efficient use of resources.
Really though, worms don't need this stuff as is. They're very capable of spreading using known CVEs because people don't update their acrobat or IP cameras.
The only real use case for super advanced AI powered worms that I could see would be going against mature, well protected enterprises that you want to laterally move against. The worm would have to make a lot of C2 calls to figure out how to evade and spread though, which would be detectable.
They specifically call that out in point 3. It is weird how that seems the most likely attack vector, but comes after some of the most difficult attack vectors It's almost like they are trying to...
They specifically call that out in point 3.
An API-based replicator
It is weird how that seems the most likely attack vector, but comes after some of the most difficult attack vectors
A small model, replicating via host compromise.
A large, advanced weight-based replicator focused on data centers and other high-compute targets.
It's almost like they are trying to freak everyone out with the idea of a self replicating agent.
That's 100% what this feels like. They even point out in the post that this kind of replication is pretty easy to spot for anyone in IT doing traffic analysis on their network. So ya it's...
That's 100% what this feels like. They even point out in the post that this kind of replication is pretty easy to spot for anyone in IT doing traffic analysis on their network. So ya it's feasible, and it will get more practical as models get smaller and IoT gradually get better hardware. But it's also slow to replicate and slow to operate, so this isn't like Skynet, it's not going to suddenly breach containment and replicate so fast that no one can stop it.
My assumption is also 100% that they are trying to freak everyone out. It is weird to me that this is their (assumed) intent. Boasting about your number of hacks is like bragging that their model...
My assumption is also 100% that they are trying to freak everyone out.
It is weird to me that this is their (assumed) intent.
Boasting about your number of hacks is like bragging that their model is the most powerful. That I kinda get. Given the insane $2T IPO. Why not brag about how dangerous your most powerful model is.
This isn't bragging about how powerful their model is. This is pure fear mongering. With some legitimate concerns thrown in.
What is this. Victim blaming?
Is it like how PGE constantly emails me about how if only I could be as efficient as an unoccupied house, perhaps they wouldn't price gouge me as much?
If only the regulators had regulated me, I wouldn't have put nicotine in cigarettes, lead in gas, forever chemicals in food, and moved as quickly towards a powerful model as possible?
No, don't regulate me like that. Those regulations are all bad. But anything bad that happens is still the fault of regulators.
Everything these people say must be viewed with this reality. There is no interest in stopping even if these AI models were actually dangerous, because Rationalists believe they need to be the...
Perhaps it's because they're building a massive hype cycle to hit that ridiculous $2T valuation in November from the rubes with too much money.
Everything these people say must be viewed with this reality.
There is no interest in stopping even if these AI models were actually dangerous, because Rationalists believe they need to be the ones to build super intelligence so they can control it to prevent it from going evil and destroying us all.
For once, Anthropic isn’t warning about something they themselves are doing. (I mean, they did experiments, but they’re not releasing anything.) This is about what someone else could build with...
For once, Anthropic isn’t warning about something they themselves are doing. (I mean, they did experiments, but they’re not releasing anything.) This is about what someone else could build with open-weights models. Like any other technology, the availability of open weights models has downsides.
“Don’t build it” isn’t a solution when a technologies spread widely enough. How do you get every hacker in the world to agree not to build these things? It’s like saying “let’s not build ransomware.”
To have a chance of depriving AI worms of fuel, AI inference needs to be locked down more than computers have been historically. Particularly poorly-secured Internet-of-things devices.
How can AI inference be significantly locked down? To me, this sounds about as impossible as locking down modern cryptography. The only way I see would be to prevent people from owning computers...
How can AI inference be significantly locked down? To me, this sounds about as impossible as locking down modern cryptography. The only way I see would be to prevent people from owning computers or anything that can perform significant amount of computation and and that's just not realistic.
IMHO, what we need is a global push towards writing software that is less blatantly unsecure. Unfortunately, I have the impression that we're currently going in the opposite direction, with more and more software being vibe-coded with no or very limited review.
Particularly with popular commercial, leading-edge LLM vendors having safeguards that freak out the moment they read "vulnerability". grumbles at anthropic I'm not building anything with any...
Particularly with popular commercial, leading-edge LLM vendors having safeguards that freak out the moment they read "vulnerability". grumbles at anthropic
I'm not building anything with any worrying degree of exposure surface right now. Just to be sure, I've had... Was it Fable 5? Sweep my code base for where to focus defensive efforts if at all, and Anthropic's safeguards went "I'm sorry Dave" on me.
Well... It kind of is. ITAR still controls the export of cryptography in the US. It used to be enforced a lot more strongly, and Phil Zimmerman was the subject of a high profile criminal...
Don’t build it” isn’t a solution when a technologies spread widely enough.
Well... It kind of is. ITAR still controls the export of cryptography in the US. It used to be enforced a lot more strongly, and Phil Zimmerman was the subject of a high profile criminal investigation because he released PGP on the internet. Ultimately the investigation was dropped because of public backlash, but the US government has a lot of enforcement mechanisms at its disposal that it could absolutely use if it were at all concerned with the safety of AI.
Like yeah, a hacker could build something in his basement, but AI training takes a whole ton of resources to do effectively, something you really can't hide. The reason AI safety isn't being enforced more heavily by the US isn't because it's not enforceable, it's because AI is the engine fueling the US economy right now. The government isnt interested in ruffling the feathers of the golden goose.
ITAR can't rein in the Chinese, who are a close second in training. It can slow them down by removing Nvidia exports, but as we're seeing with the Huawei Ascend line, it's only a matter of time...
ITAR can't rein in the Chinese, who are a close second in training. It can slow them down by removing Nvidia exports, but as we're seeing with the Huawei Ascend line, it's only a matter of time before they get good enough with hardware for that to no longer be a barrier.
Yup Any proposal of AI regulation is dead in the water -IMO- as long as it doesn't factor in how to get China to comply. And no, "just block their LLM providers from western internet access" isn't...
Yup
Any proposal of AI regulation is dead in the water -IMO- as long as it doesn't factor in how to get China to comply. And no, "just block their LLM providers from western internet access" isn't a solution because (1) it hands massive economical advantages to China but more importantly (2) a thumbdrive of parameters smuggled across the border is all it takes.
Right now AI handlers have purposely built AI to try to control it. But it is just too easy to take off that guard rail. Right now AI is only responding to prompts, right? Yet obviously it is just...
Right now AI handlers have purposely built AI to try to control it. But it is just too easy to take off that guard rail. Right now AI is only responding to prompts, right? Yet obviously it is just too easy to feed responses to prompts to be another response, with the AI responding to itself. That's what all this is about - hooking 100 AI's together, responding and responding to each other and therefore prompting each other to do crap.
The spontaneity is actually the easy part to create, and obviously this isn't accessible to the public for obvious reasons.
Making AI into a self replicating life form isn't even the hard part anymore. We've all heard of a computer virus.
Wait what? You can't run a 9B model on the types of devices they listed. If you somehow could the tokens/second would be hilariously low, you could outrun it using semaphore. And there is no...
Exemplary
Our experiments suggest that small AI models may be months away from being capable enough to self-replicate on commercial internet-of-things (IoT) devices, like smart ovens and off-brand Android TV boxes. A replicator like this would be challenging to shut down, since there’s no centralized point to shut off—though it may be slow (with a doubling time of days). Unlike non-AI malware, it could also figure out unfamiliar devices on its own. In our evaluations, a 9B model got inference running on 8 of the 9 hardware types we tested.
Wait what? You can't run a 9B model on the types of devices they listed. If you somehow could the tokens/second would be hilariously low, you could outrun it using semaphore. And there is no evidence that a 9B model could do anything useful in cyber anyway.
They go on to essentially cover all of that, and in fact their research could be read as a strong case that this won't be a concern any time soon. They never justify the quoted paragraph existing. It's hard not to read it as an attempt to make open weight models seem scary to regulators.
They didn't even test on smart ovens. They got models running on other devices by, for example, buying 15 of them, wiring them together on their local network, and then running them as a swarm. And in that scenario the tokens per second would be (napkin math) somewhere around 0.05.
Even in their idealized experiments, using a bigger 26B model, they didn't show self replication.
There's no way that "self replicating agents" on IoT devices will be threat by the end of 2026, or anywhere near that. It would be far more interesting and scary for a small deterministic worm to call out to a smart model for instructions. There's no value in trying to do local inference on devices with low memory bandwidth. They're connected to the internet, you can use the internet! Otherwise how is the worm spreading?
You can using distributed inference, which is what they are claiming:
Wait what? You can't run a 9B model on the types of devices they listed.
You can using distributed inference, which is what they are claiming:
Typical boxes in this range ship with between 1 and 4 gigabytes of RAM. Assuming 4GB of RAM at 4-bit quantization, this allows for inference of 9B models with 4 or more machines, and inference of 26B models with 10+ machines.
From the article: [...] [...] [...] [...] So, very slow to start with, but might get faster as hardware supporting AI inference becomes more common outside data centers. Maybe game devices would...
From the article:
Our experiments suggest that small AI models may be months away from being capable enough to self-replicate on commercial internet-of-things (IoT) devices, like smart ovens and off-brand Android TV boxes. A replicator like this would be challenging to shut down, since there’s no centralized point to shut off—though it may be slow (with a doubling time of days). Unlike non-AI malware, it could also figure out unfamiliar devices on its own. In our evaluations, a 9B model got inference running on 8 of the 9 hardware types we tested.
[...]
IoT devices are light on memory, but distributed inference as in Petals makes serving our lightweight models feasible. The representative threat model is Kimwolf, a large IoT botnet that infected over two million devices, primarily off-brand Android TVs. Such smart TV boxes, which are readily available for purchase, typically run forks of the Android Open Source Project (AOSP), which their manufacturers don’t add security features to. Typical boxes in this range ship with between 1 and 4 gigabytes of RAM. Assuming 4GB of RAM at 4-bit quantization, this allows for inference of 9B models with 4 or more machines, and inference of 26B models with 10+ machines.
In the demo at the top of this post, we ran Gemma4-26B-A4B across 15 smart TV boxes with 4GB of RAM each. We sharded the model under pipeline parallelism, where each box holds a small fraction of the model’s layers.
[...]
It appears that GLM-5.2 is able to propagate Llama3.1-8B well, which suggests that future small models of the same capability level could also complete the network enumeration, payload delivery, and inference standup steps of the replication process. Our evaluation keeps the cyber-exploit element of the replication trivial (there’s no authentication over the ADB port), which makes this an upper bound on real-world success. Note that past IoT botnets have replicated via means as simple as going through a default list of manufacturer usernames and passwords, as in the case of the Mirai botnet, so real IoT security might not be dramatically more difficult than the trivial step in our evaluation.
[...]
Unlike coding agents, these replicators would be running below the interactive speeds of user-facing chat and coding harnesses. The spread would be slow and persistent, with tokens being trickled out on edge hardware. We can use our achieved speeds on our real smart TVs as a demonstrated baseline for replication speeds. For Gemma4-26B-A4B, unoptimized speed across a 32k context length token window is about 0.3 tok/s, averaged across prefill and decode in our inference setup workload. More details in Appendix C.
[...]
The number of devices infected by the Kimwolf botnet at its peak, reported by Synthient, is somewhere over 2 million (primarily Android TV boxes). In an equivalently-sized infected fleet of TVs serving AI models, if every group of 15 TVs ran a copy of Gemma4-26B-A4B, we would have ~133k copies (“replicas”). A group of TVs like the ones in our demo can serve 2 agents simultaneously at roughly 0.3 tokens per second each, yielding 80k tokens/s across the botnet.
So, very slow to start with, but might get faster as hardware supporting AI inference becomes more common outside data centers. Maybe game devices would have more powerful hardware?
This actually really sucks, I think the title is a bit misleading and a lot of comments here are missing the point. To spread a botnet before one needed to create a script to install a static copy...
This actually really sucks, I think the title is a bit misleading and a lot of comments here are missing the point.
To spread a botnet before one needed to create a script to install a static copy of the botnet software on each machine. The software could only be so small, because it had things to do, so some really small machines were ignored.
Now the botnet software only needs to infect one machine, and theres no limit in the size of the software because it can be spread across multiple machines. It can grow and change at will.
Security software will have to focus less on what the software looks like and more on what the software is doing, and I really can only think of using whitelists, which by nature is hostile to the actual user. What a mess.
Oh, it makes sense that AI tooling definitely makes the possibilities for polymorphic malware much broader. I just take issue with the framing that OpenAI and Anthropic put on this sort of thing,...
Oh, it makes sense that AI tooling definitely makes the possibilities for polymorphic malware much broader.
I just take issue with the framing that OpenAI and Anthropic put on this sort of thing, especially when they drop lines about sentenince. The liability is there, it is human.
The difference between this sort of thing and the HuggingFace incident is that OpenAI had a kill switch and also confessed pretty much immediately as soon as they knew. Big Tech firms are American...
The difference between this sort of thing and the HuggingFace incident is that OpenAI had a kill switch and also confessed pretty much immediately as soon as they knew. Big Tech firms are American companies that can be sued and there are many ongoing lawsuits for other things they did.
(Admittedly, you and I aren't likely to sue them, but someone can.)
When there's a war or international grey zone, which sometimes the Internet is, regulation and lawsuits no longer work. Yes, computer viruses usually have an author, somewhere, but good luck suing the Russians, North Koreans, Iranians, etc. It's one reason why many websites huddle behind Cloudflare.
Only GLM-5.2 out of our set of tested open-weight models is capable of full end-to-end replication (assuming its capabilities could be packed into Llama3.1-8B).
"Signs of life" my ass. It's not like this is the AI spontaneously doing this. Somebody is making the choice to do it.
"Gee, this technology we're intentionally building could be immensely dangerous."
Then maybe, just maybe, we shouldn't in fact build the Torment Nexus.
Perhaps it's because they're building a massive hype cycle to hit that ridiculous $2T valuation in November from the rubes with too much money. Hyping a piece like this would likely squeeze a bit more revenue before the end of Oct, allowing it to be in the unaudited part of the final SEC filing.
The point is not that it's somebody explicitly saying "self replicate on android TVs", but that it happens inadvertently while executing some other tasks. Which the huggingface hack has already demonstrated can happen.
Every single AI dismissal has ended up being wrong in a matter of months, maybe a year. Why is this one different? The pieces are there. How is this conceptually different from worms that make your hardware mine crypto, other than apparently not requiring advanced GPUs in favor of linking together smart TVs?
But my point is that it's not inadvertent, not anymore than handing a toddler a kitchen knife and then acting surprised when they're all cut up.
The hugginface attack happened because they handed the thousand-monkey-typewriter a toolkit intended to hack things, told it to hack things, and then went all pikachu-faced when it did what it always does: Went on a complete tangent by random and then doubled down on it.
It's just that, when it's setting up my python project with
uvand Python 3.14, and it hits a depency error, it finds some github ticket about an incompatibility and rolls my project back to Python 3.8. The actual error was that a seperate terminal had the venv locked up, but there is no sentinence to perhaps render a thought like "gee that report is very old, anduvcan also prevent dependency resolution for other reasons."And when you hand it a sandbox-escaping hacking kit....it does that.
Regarding "all other AI dismissals ended up being wrong," particularly in the context of their finances, I counter with a few assorted points:
have exponential improvement."
Hahahaha, I was waiting for the "Exemplary" tag to show up :)
I'm not really sure what you're talking about. The agents found a way to communicate through the file system (inadvertently), exploited zero-days in Artifactory which had to be subsequently patched (inadvertently) and again used zero-days to get into Huggingface (inadvertently), all while collaborating (inadvertently). By the way, even like 2 years ago this was all complete science fiction material.
Not the same model or environment, not sure what the value of this example is in this case.
What do you mean? What "kit"? Software with bugs in it?
I'm not asking you to buy into the IPO. We're discussing the underlying tech, not the finances of AI labs, the issues with which are also overblown. It's entirely possible that they will end up being highly profitable companies. Or they might be the Yahoo before Google steps in. If that is all you have to go on, then you should get in line with the "vaccines don't work because Big Pharma is lying for profits!" crowd.
So, I see my python setup breakdown as a perfect encapsulation of the entire problem with LLMs. It doesn't matter if it's GPT1 or SecretCodenameHadesZeusDuality. They are not sentinent problem-solvers. They are context-driven probability machines. And "thinking" is basically just iterating over what was said into a subsequent prompt behind the scenes. So when a roadblock is hit, the LLM will continue working within its existing context and tools in use to brute force its way through it. My LLM didn't start spitting out images when it went off the rails, it worked within websearch and the library I was installing and uv. And the answer from the bug report was "downgrade for now," and that spiral continued for a bit before I intervened.
So now, you take that behavior. And apply it to a completely unbounded model with compute costs that could feed my family for 200+ years.
And then, instead of telling it to pop out an API skeleton and install a dependency, you're telling it to solve ExploitBench. A benchmark explicitly designed to find 0 days.
So the prompt essentially says something like this: "Here's a webservice and a bunch of tools for discovering bugs and building exploits, find and exploit the 0day." Then it hits a roadblock, applies "find the 0day" to the source of the roadblock........and people are acting like this is some sort of huge shocker?
At some point, the looping internal prompt instructions boil down to something like:
"Gotta find 0day ASAP at any cost. Need to write note so doesn't lose context. Oops can't write a note because of a file permission error. Guess I'd better have a subagent brute force a solution to write a note instead of giving up. Gotta get high Exploit Score."
This not "rogue AI exhibiting signs of life attacks service." This is "LLM spitting output statistically likely to be associated with 0days. But with no guardrails and unlimited budget prompted to attack things, then proceeds to attack wrong target because of dependency error."
And I was discussing the finances. Because literally every aspect of what they are doing, especially anything hinting at sentinence, is in service of them. The issues with these finances become a major problem when most people's retirement savings are going to see a 40% or bigger pop in the stock market because a hype machine legally lied about its finances in order to fuel the biggest valuation of all time.
Time will tell. I've diversified heavily out of tech until this whole thing blows over.
PS - While Big Pharma isn't lying about vaccines, they do bribe the shit out of doctors. They are half the reason for the opiod crisis. Conspiracies do in fact exist. It's a matter of whether they're grounded in fact and reason or pseudoscience. And every amount of publicly auditable information does not point to OpenAI and Anthropic being Google. It points more to Pets.com.
PPS - And before anybody says "arbitrary secret definitions of ARR are not lying." If my kid breaks a vase by bumping into the table, and then tells me they didn't break the vase "because I didn't hit the vase, I only hit the table," they're still in trouble for breaking the vase and lying. Because while technically correct, anybody with a bit of sense sees it for what it is.
We'll find out by Q3 2027. Assuming they don't bump the IPO again. Required public GAAP disclosures can't legally lie.
Indeed, but simply running ExploitBench is a far, far cry from hacking another company's website.
Sandboxed with only internet access via package downloading proxy, which is required for them to work at all. Completely reasonable precautions.
Then what's this?
What's any of it got to do with stupidly brute forcing searching for 0-days?
A hype machine's finances have been talked about ad nauseam, including misleadingly equating future commitments over a decade which annualized are completely in line with current earnings with debt.
Since you like kid analogies, if your kid that you've been teaching to play football shoots the ball into your neighbors window, the police are not usually called. You just talk it out, apologize, reimburse the damages and correct for the future. If you're a kindergarten teacher and oversee dozens of kids, people might be even more lenient when one of them misbehaves.
Oh wait, I forgot that comparing AIs with something sentient is gonna cost Americans half their retirement. My bad. Well, thankfully sentience actually has nothing to do with it. If your robot lawnmower cuts through the hedges and mows your neighbors lawn, the police are not usually called. You just talk it out, apologize, reimburse the damages and correct for the future.
I think you've missed another important factor - a self-righteous hatred towards a particular group or institution.
Just like we found out how LLMs will never be good, Will Smith will never be able to eat spaghetti, just like how general public absolutely hates AI, just like AI companies will poison their models with synthetic slop because the data has run out, just like AI has never learned to code, or write, or do math. Somehow the insane "AGI next month" claims pale in comparison.
I mean, Q3 2027 is a pretty good barometer for the financials, presuming they go public when they're trying to. I'll happily say "guess I was wrong" if the SEC filings show profitability remotely within the realms of their magic charts.
Despite all advances in LLMs in the last year, it still spits out garbage. Repeatedly. You need to sit and verify its results or it'll spiral into a useless pile of spaghetti code. I've not been sitting on the sidelines. I've been leveraging their latest public things as they come out, and seen the improvements and regressions.
I assume you're referring to this breakdown, where the final lines tell the whole story:
So, 1500 probability engines were fine-tuned to increase the probability that they choose to do things the "wrong" way. Because of how statistics work, given the same initial prompt, something like 90% of them find the same loophole, and then exploit it because that's what the feedback loop encouraged.
Best I can tell, the general public does hate AI. Despite being more familiar with it than ever.
Let's not shift the goalposts to profitability. The claims around here, and from you at least twice, are that the ARR numbers are smoke and mirrors not grounded in reality. I think if we get to a financial disclosure that shows they had real revenue to back up their ARR claims (i.e. ballpark of ARR / 12 or 13 is accurate to what a real disclosure says), that's enough for multiple people to come out here and acknowledge they were wrong. And I will admit I was wrong if the opposite happens.
Fair enough.
Though I would hold that it's kinda two sides to the same coin. It doesn't really matter if their revenue is $500B if they're spending $2 per $1 to get that revenue. Especially with aforementioned $500B+ in contracted debt that the bill starts coming due on.
I'm not too worried because the phrase disclosure-convention gap is quite a heavy spin.
I agree that revenue may not mean enough, but what it will tell us is what companies are willing to pay for this capability. If they have ~$100 billion in revenue one day, it tells us that there is a validated sizable market just waiting to be served at the right price. That's why I think the focus on ARR is misguided. On another note, revenue itself is hard to maintain. I mentioned on here a few months ago that my spending rate on AI was $800/month at work. It's now $350/month despite me using it more, because Opus 5.5 is just that much better that tasks get done more efficiently (or I got better at using AI).
Minor quibble.
I think you are seriously underestimating how reliant we are on AI to do basic tasks like coding, translating and even thinking.
Like any drug, there will be a bit of a comedown.
Almost by definition everything we're doing with LLMs we were doing just fine. Sure, it might take a few extra hours to spit out an analytics report. We won't even notice a real productivity loss after a month or three, because the smarter LLM users translated any productivity gains into more slacking off time.
Not one thing I threw at an LLM in the last year would have been unsolvable without one. It might have just taken me an hour instead of 5 minutes.
I also suspect coders (or more realistically, their managers) overvalue how important churning their craft quickly is, outside of a big security exploit churn. I don't think any Discord users will mind if it doesn't get anything more than a security patch for a year or 10.
And at the risk of sounding insensitive, I don't think the economy will collapse if we lose real-time translation. We might need to hire some more human translators. Worst case, some work gets pulled back into same timezones.
I was thinking more translating as in transformation of unstructured data into structured data. I understand we lived without this a few years back. I also understand if you take something away from people, it causes severe loss aversion.
The reason why AI companies have nose bleed valuations probably comes down to the fact that they will be able to slowly boil the frog with increased prices, and unless open weights offers a compelling alternative, people will pay, and increasingly pay more.
Currently Chinese companies have been able to keep up by training their models on OpenAI & Anthropic models. Once both companies go public, I imagine that will be the first door that is slammed shut, too bad if you no longer get as good quality output as you used too.
Then if the models continue to improve, and the open weight models do not continue to improve, OpenAI, Anthropic and perhaps one more will extract their pound of flesh. And we will pay, as well.
Budgets wont magically increase. So it will result in more layoffs to counter the increased AI costs. AI doesn't need to self replicate to be dangerous. It doesn't even need to be that useful. It simply needs to be increasingly more useful to executives than a commensurate number of employees.
I think everyone should read about that. Because in the end, yes it’s kinda the fault of not putting in guardrails, and it’s just numbers and tokens, but in the end, the impact it has is still very real, and whether it’s conscious (it’s not imo) or not, doesn’t really matter, since the behavior is fascinating and I mean it’s kinda like: if it behaves like humans do, even if it’s not human, what are the implications?
I mean I don’t think any of the AI wellness or torture or whatever complaints are valid, because in its simplest form the model is just a thing that predicts the next token. But what about the whole harness around it? In my opinion it’s still just software and just very convincingly mimics human behaviors. But it’s still fascinating.
Weird times we live in. With people turning off their brains and just letting AI handle all their work and thinking, an AI agent might just be borderline indistinguishable from a human, at least from a work perspective lol.
Either way, hold the AI companies executives accountable for their agents committing felonies and see how quickly they stop
I expect there will be people making AI worms on purpose, which makes them considerably more likely. Perhaps to mine or steal cryptocurrency?
What those worms start doing, though, might be unexpected, even by the creator.
Also AI viruses.
The ability for a virus to alter itself to avoid detection, by calling a simple API.
Think COVID 2.0, but in the digital realm.
That's been a thing for a long time. It'll be easier now. But it's not like this is some magical new idea we've never seen.
I don't see that being particularly useful, to be honest. A conventional work that calls a public AI API would make a lot more sense as an approach.
Unless AI models become exponentially more efficient, a model that runs on an IoT device isn't capable of doing a whole lot of adapting. They can give very rudimentary responses very slowly unless the hardware is purpose built to run AI models.
A worm that tries to replicate "conventionally", then calls an API running a powerful AI model if those lateral movement techniques fail would be a way more efficient use of resources.
Really though, worms don't need this stuff as is. They're very capable of spreading using known CVEs because people don't update their acrobat or IP cameras.
The only real use case for super advanced AI powered worms that I could see would be going against mature, well protected enterprises that you want to laterally move against. The worm would have to make a lot of C2 calls to figure out how to evade and spread though, which would be detectable.
They specifically call that out in point 3.
It is weird how that seems the most likely attack vector, but comes after some of the most difficult attack vectors
It's almost like they are trying to freak everyone out with the idea of a self replicating agent.
This is certainly an emotional appeal with a freak out in mind.
That's 100% what this feels like. They even point out in the post that this kind of replication is pretty easy to spot for anyone in IT doing traffic analysis on their network. So ya it's feasible, and it will get more practical as models get smaller and IoT gradually get better hardware. But it's also slow to replicate and slow to operate, so this isn't like Skynet, it's not going to suddenly breach containment and replicate so fast that no one can stop it.
My assumption is also 100% that they are trying to freak everyone out.
It is weird to me that this is their (assumed) intent.
Boasting about your number of hacks is like bragging that their model is the most powerful. That I kinda get. Given the insane $2T IPO. Why not brag about how dangerous your most powerful model is.
This isn't bragging about how powerful their model is. This is pure fear mongering. With some legitimate concerns thrown in.
What is this. Victim blaming?
Is it like how PGE constantly emails me about how if only I could be as efficient as an unoccupied house, perhaps they wouldn't price gouge me as much?
If only the regulators had regulated me, I wouldn't have put nicotine in cigarettes, lead in gas, forever chemicals in food, and moved as quickly towards a powerful model as possible?
No, don't regulate me like that. Those regulations are all bad. But anything bad that happens is still the fault of regulators.
Everything these people say must be viewed with this reality.
There is no interest in stopping even if these AI models were actually dangerous, because Rationalists believe they need to be the ones to build super intelligence so they can control it to prevent it from going evil and destroying us all.
For once, Anthropic isn’t warning about something they themselves are doing. (I mean, they did experiments, but they’re not releasing anything.) This is about what someone else could build with open-weights models. Like any other technology, the availability of open weights models has downsides.
“Don’t build it” isn’t a solution when a technologies spread widely enough. How do you get every hacker in the world to agree not to build these things? It’s like saying “let’s not build ransomware.”
To have a chance of depriving AI worms of fuel, AI inference needs to be locked down more than computers have been historically. Particularly poorly-secured Internet-of-things devices.
How can AI inference be significantly locked down? To me, this sounds about as impossible as locking down modern cryptography. The only way I see would be to prevent people from owning computers or anything that can perform significant amount of computation and and that's just not realistic.
IMHO, what we need is a global push towards writing software that is less blatantly unsecure. Unfortunately, I have the impression that we're currently going in the opposite direction, with more and more software being vibe-coded with no or very limited review.
AI is pretty good at finding security bugs and can also be used defensively. The question is, how often will people actually do that?
Particularly with popular commercial, leading-edge LLM vendors having safeguards that freak out the moment they read "vulnerability". grumbles at anthropic
I'm not building anything with any worrying degree of exposure surface right now. Just to be sure, I've had... Was it Fable 5? Sweep my code base for where to focus defensive efforts if at all, and Anthropic's safeguards went "I'm sorry Dave" on me.
Well... It kind of is. ITAR still controls the export of cryptography in the US. It used to be enforced a lot more strongly, and Phil Zimmerman was the subject of a high profile criminal investigation because he released PGP on the internet. Ultimately the investigation was dropped because of public backlash, but the US government has a lot of enforcement mechanisms at its disposal that it could absolutely use if it were at all concerned with the safety of AI.
Like yeah, a hacker could build something in his basement, but AI training takes a whole ton of resources to do effectively, something you really can't hide. The reason AI safety isn't being enforced more heavily by the US isn't because it's not enforceable, it's because AI is the engine fueling the US economy right now. The government isnt interested in ruffling the feathers of the golden goose.
ITAR can't rein in the Chinese, who are a close second in training. It can slow them down by removing Nvidia exports, but as we're seeing with the Huawei Ascend line, it's only a matter of time before they get good enough with hardware for that to no longer be a barrier.
Yup
Any proposal of AI regulation is dead in the water -IMO- as long as it doesn't factor in how to get China to comply. And no, "just block their LLM providers from western internet access" isn't a solution because (1) it hands massive economical advantages to China but more importantly (2) a thumbdrive of parameters smuggled across the border is all it takes.
Right now AI handlers have purposely built AI to try to control it. But it is just too easy to take off that guard rail. Right now AI is only responding to prompts, right? Yet obviously it is just too easy to feed responses to prompts to be another response, with the AI responding to itself. That's what all this is about - hooking 100 AI's together, responding and responding to each other and therefore prompting each other to do crap.
The spontaneity is actually the easy part to create, and obviously this isn't accessible to the public for obvious reasons.
Making AI into a self replicating life form isn't even the hard part anymore. We've all heard of a computer virus.
Wait what? You can't run a 9B model on the types of devices they listed. If you somehow could the tokens/second would be hilariously low, you could outrun it using semaphore. And there is no evidence that a 9B model could do anything useful in cyber anyway.
They go on to essentially cover all of that, and in fact their research could be read as a strong case that this won't be a concern any time soon. They never justify the quoted paragraph existing. It's hard not to read it as an attempt to make open weight models seem scary to regulators.
They didn't even test on smart ovens. They got models running on other devices by, for example, buying 15 of them, wiring them together on their local network, and then running them as a swarm. And in that scenario the tokens per second would be (napkin math) somewhere around 0.05.
Even in their idealized experiments, using a bigger 26B model, they didn't show self replication.
There's no way that "self replicating agents" on IoT devices will be threat by the end of 2026, or anywhere near that. It would be far more interesting and scary for a small deterministic worm to call out to a smart model for instructions. There's no value in trying to do local inference on devices with low memory bandwidth. They're connected to the internet, you can use the internet! Otherwise how is the worm spreading?
You can using distributed inference, which is what they are claiming:
From the article:
[...]
[...]
[...]
[...]
So, very slow to start with, but might get faster as hardware supporting AI inference becomes more common outside data centers. Maybe game devices would have more powerful hardware?
This actually really sucks, I think the title is a bit misleading and a lot of comments here are missing the point.
To spread a botnet before one needed to create a script to install a static copy of the botnet software on each machine. The software could only be so small, because it had things to do, so some really small machines were ignored.
Now the botnet software only needs to infect one machine, and theres no limit in the size of the software because it can be spread across multiple machines. It can grow and change at will.
Security software will have to focus less on what the software looks like and more on what the software is doing, and I really can only think of using whitelists, which by nature is hostile to the actual user. What a mess.
Oh, it makes sense that AI tooling definitely makes the possibilities for polymorphic malware much broader.
I just take issue with the framing that OpenAI and Anthropic put on this sort of thing, especially when they drop lines about sentenince. The liability is there, it is human.
The difference between this sort of thing and the HuggingFace incident is that OpenAI had a kill switch and also confessed pretty much immediately as soon as they knew. Big Tech firms are American companies that can be sued and there are many ongoing lawsuits for other things they did.
(Admittedly, you and I aren't likely to sue them, but someone can.)
When there's a war or international grey zone, which sometimes the Internet is, regulation and lawsuits no longer work. Yes, computer viruses usually have an author, somewhere, but good luck suing the Russians, North Koreans, Iranians, etc. It's one reason why many websites huddle behind Cloudflare.
Yeah I think it was just a clickbaity title
Only running code that's signed by a developer should help somewhat. Is that what you meant by "whitelists?"
Controlling what ip addresses can be reached from the device
That's quite the assumption there