Patrick Boyle (a finance youtuber who apparently has a day job in finance as well) discusses the recent hubbub about AIs escaping, forming a botnet, taking over the internet, etc. etc. therefore...
Patrick Boyle (a finance youtuber who apparently has a day job in finance as well) discusses the recent hubbub about AIs escaping, forming a botnet, taking over the internet, etc. etc. therefore the US government should regulate them into slowing down. Broadly, he's taking the cynical perspective that these flames are being fanned by execs attempting to position themselves better to profit from the situation, instead of taking their talking points at face value and assuming that four billionaires are pleading to the world to rein them in, out of a reverence for the human spirit that they recently discovered. Perhaps in a Grinch-esque Christmas stealing affair where their hearts grew several sizes overnight.
Full disclosure: I'm not a fan of Patrick Boyle, as he oversteps his knowledge areas a lot, but he's mostly summarizing articles written by other people in this case. It isn't a short video by any stretch of the imagination (~30 minutes), but it isn't dense enough to merit one's full attention, in case you're looking for something to leave on in the background.
Is there any actual evidence for the widespread assumption that the people making the warnings aren’t sincere? For some arguments against, see this article.
Is there any actual evidence for the widespread assumption that the people making the warnings aren’t sincere? For some arguments against, see this article.
Yea, I find it rather off putting that there is large group of people who seem rather quick to conclude that the people closest to this technology don't know it the best or can't be trusted to...
Yea, I find it rather off putting that there is large group of people who seem rather quick to conclude that the people closest to this technology don't know it the best or can't be trusted to give observations. It's very obviously not just CEOs or even limited to people who stand to gain monetarily. Not to say we should trust everything they say or anything but I mean, there are a lot of signals to ignore these days and it's getting harder to credibly ignore them.
I haven't watched the video but I will, I enjoy Boyle's videos when I've seen them surfaced. I think he's probably making an honest argument and I'm all for splashing some cold water sometimes. However, there is definitely an ongoing attempt to discredit AI safety at the moment, see e.g. the 'sex cult' stories or the broad stroke attempts to paint EAs and rationalists as poorly as possible.* If you want to start chasing money and power shaping a narrative, I would probably start there.
* And to be fair yes there are a lot of weird people who are easy targets out there but the venn diagram of AI safety people and LW-types is nowhere near a circle.
I'm not entirely convinced they do. Demonstrated by the lack of safeguards in the sandbox through which the hugging face hack occured. The whole story was wild in all the wrong ways IMHO. I would...
the people closest to this technology don't know it the best or can't be trusted to give observations.
I'm not entirely convinced they do. Demonstrated by the lack of safeguards in the sandbox through which the hugging face hack occured. The whole story was wild in all the wrong ways IMHO.
I would not be surprised if at this point, nearly every part of this is vibe coded, orchestrated by agents or researched by LLMs. Which gives me the impression that AI development is more akin to fueling a hydrothermal vent, hoping superintelligence pops out.
I agree with this, but I think the situation is nuanced. The public mouth pieces of the big AI companies (Altman, Amodei, various engineers and project leads) have an established history of...
Yea, I find it rather off putting that there is large group of people who seem rather quick to conclude that the people closest to this technology don't know it the best or can't be trusted to give observations
I agree with this, but I think the situation is nuanced. The public mouth pieces of the big AI companies (Altman, Amodei, various engineers and project leads) have an established history of predictions that turned out to be wrong, exaggerations and general hype.
It makes sense to question what they say, even if they genuinely believe it, it might still be wrong.
Personally I have yet to see anything close to the capability to be truly, autonomously, dangerous in the frontier models. They can do dangerous things when directed or misused or deployed badly, but so can a lot of technology.
The current wave of fear seems to be based in the idea that, any time now, they will achieve recursive self improvement and then they will make themselves dangerous really quickly. Except there's no evidence of recursive self improvement either. I mean, put an agent in charge of its own memory system and watch how long it takes before it writes bad judgement to the record and then compounds it by building more bad judgement on top, and and so on. If you give it enough time, it will make itself worse, not better.
That's of course not how recursive self improvement would work, instead it would be agents fully in charge of the training pipeline, but the point is that their judgement isn't good enough. They would make bad decisions, which would compound over time. They'd also make good ones, and maybe end up improving themselves on average, but not faster or cheaper than a human led process.
The doomsday fear all rests on the idea that this will change, and maybe it will, but we don't appear to be close to that yet.
The model that helped solve the millenium prize problem began work four days after beginning training, and solved it in another four days. Apparently, this cost around 20 million USD. Compared to...
The model that helped solve the millenium prize problem began work four days after beginning training, and solved it in another four days. Apparently, this cost around 20 million USD.
Compared to the decades of human PhDs working on this, it seems both faster and cheaper.
We have no idea if AI companies are already putting models in charge of training orchestration. But we do know that AI helps significantly in writing and maintaining the harnesses and architectures around the training pipeline. Anthropic deprioritised video models and emphasised coding models for this reason, and gained enough of a lead over OpenAI that the latter were forced to drop their own video models.
This is a weaker form of recursive self improvement, but it is one. And it shortens timelines until the stronger form is reached. We used to always have months between step changes in model capabilities, now it's apparently 8 days.
A claim like "we aren't close to recursive self improvement" feels to me like a claim that "10,000 infections isn't close to a pandemic". Like sure, divide that by 8 billion, that's a small number. But something is clearly underway. We're dealing with an innate exponential here. It sounds extremely shortsighted and reckless to say "it's just hype, I'll panic when 10% of the world population is infected". Even 1% there would be far too late.
That's hard to compare, considering this still took the input and direction of human PhD's working for decades on the problem to get to this point. For all I know, this problem was a 99% solved...
Compared to the decades of human PhDs working on this, it seems both faster and cheaper.
That's hard to compare, considering this still took the input and direction of human PhD's working for decades on the problem to get to this point. For all I know, this problem was a 99% solved jigsaw puzzle, and AI was able to help these people get it to 100%. Which is important, but not "we saved decades of work/talent" levels of savings,
It kind of reminds me of the claims that AI can make Doom the 1993 video game... then you realize it did so by training it on Doom. That doesn't exactly solve the issue of game development unless you want a legally distinct Doom (and I'm unsure how "legally" that distinction is).
A claim like "we aren't close to recursive self improvement" feels to me like a claim that "10,000 infections isn't close to a pandemic".
In this case, the claims have been made for 4 years, rather than less than 4 days to spread through a major country (and less than 4 months to spread to the globe). Not to mention that such infection is this time 100% objectively spread by human intervention. Leader truly worried about such a cyber pandemic and not their bottom line can simply... stop? The handwave of "but everyone else is doing this" only serve the bottom line, not humanity's future if you truly fear this spread.
I agree that it’s hard to do a cost comparison. It seems like $20 million is quite a lot of money? And I don’t know how to account for the time mathematicians spent on the problem. But I don’t...
I agree that it’s hard to do a cost comparison. It seems like $20 million is quite a lot of money? And I don’t know how to account for the time mathematicians spent on the problem.
But I don’t think these achievements should be discounted because they’re not entirely original. That seems like too high a standard. For a human, being well-read and understanding the literature well is usually considered an advantage. Similarly, effectively building on all the mathematical research that came before doesn’t make AI any less formidable as a mathematician.
Figuring things out from scratch is a different sort of achievement because it’s working with a handicap.
Depends on the nature of the work. Overall, the idea that AI can replace decades of research implies that all the answers to these unsolved problems lies hidden in data already made, and no new...
effectively building on all the mathematical research that came before doesn’t make AI any less formidable as a mathematician.
Depends on the nature of the work. Overall, the idea that AI can replace decades of research implies that all the answers to these unsolved problems lies hidden in data already made, and no new data will ever be needed again. Which I find doubtful.
The big issue here is partially in your language here: AI didn't "build" upon the solution by itself. I doubt if you gave any other math institution not directly working on this problem the same AI, that they could solve the problem in anywhere close to 4 days/$20m of processing. To use a common metaphor, the AI was a fancy calculator being given specific inputs, and making intermediate outputs (leading to more inputs). I don't think most people would give the calculator the primary credit here.
As of now AI cannot really create new ideas out of what I've heard called "the creative void". That is human ingenuity AI has yet to replicate. AI instead is really good at synthesizing a mass amount data and revealing trends among it. And we do have a lot of data to synthesize, so this is very valuable. But conflating such synthesis to "creation" doesn't really help the larger conversation against whether AI is here to "give untold efficiency boosts to labor" vs. "replace the entire labor force and bring about a new age of feudalism".
I don’t think building on someone else’s results is necessarily all that creative, but it seems like proving a difficult math theorem is not much like using a calculator or “revealing trends.”...
I don’t think building on someone else’s results is necessarily all that creative, but it seems like proving a difficult math theorem is not much like using a calculator or “revealing trends.”
Also, other mathematicians and even non-mathematicians have found useful results using frontier AI’s. It’s not just one result, it’s many. Sometimes they share the prompt they used.
Ok then! That's not true. It began work after training was complete. It was fully cooked. I don't know if it was cheaper, but it was definitely faster. Mathematics is ever so very different from...
Ok then!
The model that helped solve the millenium prize problem began work four days after beginning training
That's not true. It began work after training was complete. It was fully cooked.
Compared to the decades of human PhDs working on this, it seems both faster and cheaper.
I don't know if it was cheaper, but it was definitely faster. Mathematics is ever so very different from recursive self improvement though. You changed the framing and then replied to your own version.
This is a weaker form of recursive self improvement, but it is one.
If that's what we're calling recursive self improvement, I don't think it's all that scary. No argument that agents are involved in various parts of the process, model labs have been talking about that for some time.
For LLMs to get out of hand they'd need to run the pre and post training process autonomously and accelerate it dramatically. Currently velocity is constrained to a significant degree by compute, so the agents would need to solve that first. Even if LLMs that smart and reliable existed (they absolutely do not), they'd still need to be given the resources to do it. It's like not these companies aren't paying attention to what's using the compute they're spending historically unprecedented amounts of money on.
If the other sort of recursive self improvement were achieved (the sort where a human just pressed go and the rest ran itself and resulted in a better model, faster, than a human led process), they'd say something. They would have just won one of the most important races in the field. It would be invaluable pre-IPO hype potentially worth billions of dollars. And, if it actually worked, they'd sell access to the resulting model (which would be both the smartest and most compute efficient model on earth). Not selling it would be equivalent to setting the 100's of millions+ they spent on training on fire. The biggest problem they'd have, with the best model on earth, would be figuring out how to handle the demand.
So we won't need to speculate, we'll be able to read the headlines. If they were smart they'd also announce a pause on recursive training while they assessed the risks. Otherwise governments would force them to take a beat just to quell the worldwide panic. People are freaking out about the possibility with no evidence that it can even be done, imagine the response to the thing itself.
It sounds extremely shortsighted and reckless to say "it's just hype, I'll panic when 10% of the world population is infected". Even 1% there would be far too late.
In this comparison, what is the pathogen? You'd need something self replicating and with a troubling mortality rate. Something observable. Otherwise it seems to me that you're just picking something scary and arbitrarily comparing it to the topic to inject emotion.
It sounds like what you're describing is how you feel about AI, which is fine, but you're framing it like a conversation about what is. So concrete, in fact, that by having a different perspective I'm shortsighted and reckless. Which is quite an accusation to make against an internet stranger.
My personal wave of fear is that we're exploiting Earth way too much, mostly for tech lords to become even more wealthy and powerful. And of course colonization of countries worldwide by tech...
The current wave of fear
My personal wave of fear is that we're exploiting Earth way too much, mostly for tech lords to become even more wealthy and powerful. And of course colonization of countries worldwide by tech corporations. I'm not afraid of LLMs becoming too powerful. I'm afraid of people behind those LLMs becoming too powerful. And afraid of too many resources spent of this idiotic dream of theirs, and the price paid by the rest of the people.
If you expand it to wealth and power consolidation, and capital otherwise corrupting systems and contributing to the destruction of our ecosystems, I'm right there with you. If it's just about...
If you expand it to wealth and power consolidation, and capital otherwise corrupting systems and contributing to the destruction of our ecosystems, I'm right there with you.
If it's just about AI... It would be wonderful if that was the biggest of our problems. Or even a big enough part that erasing it could change the trajectory.
I think one of the things that makes people suspicious is the simple observation that, despite all their warnings, these people continue to build the thing that they are warning about. I...
Yea, I find it rather off putting that there is large group of people who seem rather quick to conclude that the people closest to this technology don't know it the best or can't be trusted to give observations.
I think one of the things that makes people suspicious is the simple observation that, despite all their warnings, these people continue to build the thing that they are warning about.
I understand that they have their reasons (including, but not limited to, uh, money), but it's very hard to convince someone that you are sincere when your actions appear to contradict your words.
A big part of the danger, as I understand it, is exposing and making it relatively easy to exploit software security vulnerabilities at rate not anticipated by standard security practices of the...
A big part of the danger, as I understand it, is exposing and making it relatively easy to exploit software security vulnerabilities at rate not anticipated by standard security practices of the last several years or decades. Standards can adapt, vulnerabilities can be patched, and if you as the "good guy" give friendly actors tools and information that helps them make that transition, then "your side" will be at an advantage. Compared to stopping building of the thing, the "bad guys" get the tools first, and "your side" suffers. I don't think unilateral stand down would make sense to the people involved in AI development, even those whose fears of the technology are sincere.
I don't think that even if people are giving credit to the idea that all of these dire warnings from AI execs are greed-induced faux hype for their AI products that it means people are...
I find it rather off putting that there is large group of people who seem rather quick to conclude that the people closest to this technology don't know it the best or can't be trusted to give observations.
I don't think that even if people are giving credit to the idea that all of these dire warnings from AI execs are greed-induced faux hype for their AI products that it means people are discrediting the actual threat of AI. I think both things can be true, that they're not to be taken at face value for what they're saying but that what they say could also be true. They are untrustworthy in the sense that they have very conflicting incentives.
There are dire safety concerns with AI, but the 'solution' being offered is sort of creating this narrow focus of it being the only way to handle it. Whether it's 'slow it down' or some similar angle, all of this is an inevitable product of relatively unrestrained capitalism. Instead of the solution being to rein AI in, maybe it's to rein capitalism in. Unrestrained capitalism is a race to the 'end', so is it any surprise that we're now getting all these signals from people in high positions proclaiming something resembling an 'end'? The reality is that the people with the resources making these signals don't want to risk or give up any of their resources to attempt to resolve the problem. They want to continue to hoard their resources and hoard their power while claiming to be the prophets foreseeing a problem that they're perpetuating while pretending to offer some kind of solution that doesn't involve acknowledging that they themselves are part of the problem.
Also, the 'slow it down' suggestion isn't even a solution. Slowing it down doesn't prevent anything, and I have absolutely no faith in the current US government, influential people/leaders, the general populace or the direction things have been going for the past couple decades that even if we gave them more time to react to the coming disaster that they would appropriately respond to it in a meaningful way that helps people.
Instead what they are proposing is a 'controlled fall', because it is what best helps them maintain the system and control over that system. If you fall in an uncontrolled manner, you might fall hard and fast and with significant collateral damage, of course they don't want this because the blowback could come on them. Instead in a controlled fall, they can attempt to ensure the system that best lets them hold their resources and power has the least amount of collateral damage to impact them. Why exactly should there be a fall at all? How did we get to a point where a 'controlled fall' is the solution? Because the people proposing it and the cloth they are cut from led us to this situation.
Oh, uh ... I've definitely run across some stuff where he's really off-base, and this might be one of them for some folks. I think his financial analyses are normally pretty good, for context, but...
I enjoy Boyle's videos when I've seen them surfaced.
Oh, uh ... I've definitely run across some stuff where he's really off-base, and this might be one of them for some folks. I think his financial analyses are normally pretty good, for context, but this is ... not that.
The reason I have a hard time trusting the people closest to the technology when they claim it to be dangerous is because of our country's history with snake oil salesmen. It is very good...
The reason I have a hard time trusting the people closest to the technology when they claim it to be dangerous is because of our country's history with snake oil salesmen. It is very good marketing to explain how dangerous something is right before trying to sell control of said thing. "It's a monster, it will destroy your lively hood, unless of course you buy in now! Then you get to control the monster!". I know I'm making a somewhat unsophisticated argument against advanced technologies but the sales pitch hasn't changed since the first hair growth tonic: over sell the capabilities, make the capabilities sound as scary or exciting as possible, make sure to make those not buying what is being sold feel like they're being left behind in the dust, or worse: disenfranchised through non participation. So yeah, I tend to lean toward skepticism.
Can I accurately mind-read the frontier leaders? No, I cannot. So I don't think I can ever truly "prove" that they are lying/truthful in their sentiment. I'm going to sleep so I haven't watched...
Is there any actual evidence for the widespread assumption that the people making the warnings aren’t sincere?
Can I accurately mind-read the frontier leaders? No, I cannot. So I don't think I can ever truly "prove" that they are lying/truthful in their sentiment.
I'm going to sleep so I haven't watched the video yet. But I'm sure it gives some support to that notion. For me it's extremely simple: The actors leading these frontier models are untrustworthy and they have a history of overly exaggerating, spreading misinformation, lying,
and in a few extreme cases: coaxing a genocide. They've not only burned any benefit of the doubt to be given, but are fundamentally people I do not want influencing larger policy.
See my other comment. I think this counts as “people are even worse when talking about groups of people.” If you discount the leaders, maybe engage more with what less prominent AI researchers are...
See my other comment. I think this counts as “people are even worse when talking about groups of people.”
If you discount the leaders, maybe engage more with what less prominent AI researchers are saying?
I don't know any of these people personally, so I can only judge by actions. Maybe tomorrow I can find some more solid reporting on these people's histories. But my emotional sentiments made above...
I don't know any of these people personally, so I can only judge by actions. Maybe tomorrow I can find some more solid reporting on these people's histories. But my emotional sentiments made above were supported by various reporting I've read over the years. It's not some shallow gut feeling driven by a shallow dismissal of AI or their looks or something.
I'm not a practiced speaker, though; I don't just have the months/years of various blogs, youtube videos, newspaper reports, and court testimonies available for me to display to you on demand. It'll take time to gather them up once again.
I cannot talk for Patrick Boyle, but all I said on the matter is "I do not trust these frontier leaders". At this point I don't see why I need to make more argument if this conversation is simply...
I cannot talk for Patrick Boyle, but all I said on the matter is "I do not trust these frontier leaders".
At this point I don't see why I need to make more argument if this conversation is simply going the direction of "but don't say anything without proof". I'll just leave it at that and you're free to (dis)trust whoever you want.
This is smearing together the various actions and histories of different people into a sort of collective guilt. Whatever Musk or Zuckerberg have done has no bearing on some other leader’s honesty...
The actors leading these frontier models are untrustworthy and they have a history of overly exaggerating, spreading misinformation, lying, and in a few extreme cases: coaxing a genocide.
This is smearing together the various actions and histories of different people into a sort of collective guilt. Whatever Musk or Zuckerberg have done has no bearing on some other leader’s honesty because they are different people.
It’s like assuming some politician you never heard of must be corrupt. You don’t have to trust a stranger and it’s not wrong to say that corruption is common based on an average tendency, but it’s quite another thing to assume they’re lying because they must be like the others.
You’d never want a jury to come to a conclusion because “he’s just like the others.” A journalist reporting on crime would be no good if they mixed up the details of different crimes. But that’s what we often do when we talk about people in groups.
I'm curious what sort of evidence could ever be fielded that would shine a light on the nature of one's heart -- and for what purpose? Many stories can be spun in two (or more), equally truthful,...
I'm curious what sort of evidence could ever be fielded that would shine a light on the nature of one's heart -- and for what purpose? Many stories can be spun in two (or more), equally truthful, directions. I truly do not care to understand the mind of a billionaire, so whether or not they "believe" something "sincerely" is neither here nor there. All that matters is predicting the course of the storm, not its take on AGI/ASI.
I'll dissect the article you linked tomorrow if that's OK, since it's playing some funny games with its rhetoric (it overstates an opponents position, defeats an unimportant detail of the opponent's argument, then presents a moderate position in response to their opponent's portrayed absurdity) and I'm a little too tired to explain how it's twisting reasoning in order to address a different argument while seeming to consider both sides.
I think the closest thing to a citation that the video references is David Sacks tweeting about the subject. He doesn't mention regulatory capture, just that (1) if these guys genuinely think that they're going to destroy the world, they can just stop, since (2) nothing the US government will stop the Chinese market anyways and thus we're really only talking about pacing US development of AI, and (3) positions it as forming a cartel rather than capturing a regulator. Though those two positions seem close-ish.
I think it can be quite to difficult to know anyone’s true motivations for what they do. But I’m struck by the tendency for people to simply make up a story about the motivations of strangers and...
I think it can be quite to difficult to know anyone’s true motivations for what they do. But I’m struck by the tendency for people to simply make up a story about the motivations of strangers and treat it as fact. it seems like you should at least cite something they said or something they did that seems to be revealing? Though it’s certainly possible to do that while still being quite unfair about it, it would be a start.
For example, if I wanted to make the case that Sam Altman seems rather untrustworthy, I might cite the profile in the New Yorker about him, which is based on interviews with people who have worked with him. It seems like useful background. Though it won’t necessarily reveal the exact motivations for something he said, after reading a story like that, it’s reasonable to say you know a bit about him.
But often people don’t bother to even talk about individuals in any detail. They will just smear people by association. You’re suspicious because of where you work or because of one of your company’s investors or you went to the wrong parties or whatever.
I usually don’t care to spend the time to understand people in the news in much depth, so I find it easier to simply avoid making confident statements about what they might have been thinking. It’s easy enough to share your suspicions without going so far as to assert them as fact.
People will be even worse when talking about groups of people.
Gotcha! To be clear, I'm not certain if your use of the "you" pronoun indicates that you think I'm personally invested in making some judgment on the character of these folks (as opposed to the...
Gotcha! To be clear, I'm not certain if your use of the "you" pronoun indicates that you think I'm personally invested in making some judgment on the character of these folks (as opposed to the more awkward "one", which colloquially hardly seems used these days), so perhaps it already goes without being said again, but I do not care if these CEOs' concerns are genuine or not. The only thing which matters is predicting what their endgame is, which requires a behavioural model and some assumptions about goals. Genuinely, I could not care less about their true feelings on this subject. I reject any reframe of this discourse to be centered on the personal moral character of some weirdo tech dudes.
Sam Altman is the one making the moral argument that Sam Altman cares deeply about the future of humanity ("First, we could lose control of the future to AI. This is unacceptable; we are unapologetically on Team Humanity, and AI must always serve people.", Twitter). That's clearly not the case: there are many, many more effective ways to spend your money on improving humanity's average condition than blowing it on AI development. I'd accept that it's his hobby, but citing his deep and unshakeable faith in the value of humanity during this recent corporate policy move rings hollow; where have his morals been hiding until just this moment?
Unless he's an idiot, or something, which as has been beaten to death, he almost certainly isn't.
My actual argument -- if it's OK to circle back to it now -- is that an explanation with equal predictive power is that these CEOs really, really, really like money and are taking an action which they feel best preserves their companies' abilities to make more money in the future. In practice, in this scenario, this means attempting to do some shenanigans involving the US government in order to depress development of competing companies while providing an out for any upcoming potential reduction in progress due to cash flow constraints. I have about two billion pieces of evidence that Sam Altman really, really, really likes money, and as evidence for the "shenanigans" I cite the video and that tweet again.
Re. your article in the NY times covering Altman, apologies, since as noted I'm not terribly concerned with his moral character, that doesn't feel like required reading for this conversation. I can give it a once through if you feel that it's critical to understanding your position, however.
Right! Re. the Tabarrok article, what I was alluding to was that Alex is doing the following:
Presents an indefensible position for his opponents (that the AI companies are pursuing textbook regulatory capture),
Explains why that position is indefensible (the middle section about "Regulating Business by Independent Commission"),
Concludes with a seemingly moderate position to sound reasonable ("That doesn’t prove that every rule Amodei favors is innocent of self-interest, an absurd proposition. I suspect that Amodei’s ideal may be something like a single regulated AI monopoly—safe and reasonable profitable, like the old AT&T. But that’s not the profit maximizing outcome.")
Very similar to the above issue with Sam Altman, the only person in the room citing regulatory capture and a scorched earth paperclip maximizing giga capitalist perspective is Alex Tabarrok. As noted by the topical video and (again) that linked tweet, we're discussing some generic and boring anti-competitive behaviour, not an extremely specific strategy which clearly cannot be accomplished on the timescales that're being discussed.
Yes, I didn't mean "you" in particular. Sorry about that! "Only person in the room" doesn't seem right. Which room? Who do you imagine he's responding to? I've seen many people online claiming...
Yes, I didn't mean "you" in particular. Sorry about that!
"Only person in the room" doesn't seem right. Which room? Who do you imagine he's responding to?
I've seen many people online claiming that when the leaders at AI companies asked for more regulation, it's to their own advantage, because they are attempting to exclude competition and they will somehow control the regulators to do that. (And such comments often get lots of upvotes, too!) There are people who write articles claiming this and they get shared approvingly. This position seems popular? And populist.
There are often better and worse versions of an argument, though. For any argument you can think of, there are probably people online doing it badly. So, refuting the bad version of an argument isn't wrong, but there may be more to it. It's okay to not to care that much about refuting the bad version.
Also, I'm a big believer in mixed motives. I think that's the normal state of affairs - people are often "talking their book." They stand to gain from something, but may also think it's the right thing. Neutral experts are rare. Liking money doesn't mean you can't also care about the "future of humanity" or whatever.
And besides, if some of the people making an argument are financially conflicted, there may be other people who don't have the same financial incentives. It seems wrong to dismiss an argument because some of the people who take that position have mixed motives?
So, I think pointing out financial incentives isn't wrong, but using it as a stick to dismiss arguments is. If the "wrong" people are in favor of something, that's reason to be suspicious, but not enough in itself to take the opposite side. The "wrong" people can sometimes be right about things.
np! It's text chat; this medium obliterates nuance and context. You and I? Your previous message had linked to Alex's article, presumably because it was relevant to the topic at hand. My take is...
Yes, I didn't mean "you" in particular. Sorry about that!
np! It's text chat; this medium obliterates nuance and context.
"Only person in the room" doesn't seem right. Which room? Who do you imagine he's responding to?
You and I? Your previous message had linked to Alex's article, presumably because it was relevant to the topic at hand. My take is that it:
isn't relevant (I'm not concerned with sincerity, only results),
is rhetorically dishonest (see above comment),
isn't referring to any position I'd brought up.
Re. 3, quoting specifically:
Yet when Dario Amodei says AI is dangerous, perhaps even an extinction risk, some people conclude he must be running a marketing campaign. That is stupid.
[...]
[A single regulated AI monopoly] is perfectly consistent with self-interest playing a role; it is not consistent with the bastardized theory that profit maximization is the only thing that matters or that “capture” is universal.
^ I'm not arguing that "profit maximization is the only thing that matters". You aren't (afaict). Sacks isn't. Boyle isn't. So the hypothetical entity complaining about this isn't present in the conversation to defend themselves, hence "not being in the room".
(And such comments often get lots of upvotes, too!) There are people who write articles claiming this and they get shared approvingly. This position seems popular? And populist.
No comment on most of this, but fwiw I upvote comments and posts when they're (1) well reasoned, and (2) not aggressive (or regressive, in cases where people are championing the rights of bigots or whatnot). Which means I upvote a tonne of stuff I don't necessarily agree with, just that I appreciate was written.
For any argument you can think of, there are probably people online doing it badly. So, refuting the bad version of an argument isn't wrong, but there may be more to it. It's okay to not to care that much about refuting the bad version.
Hrm. Totally fine for that to be your perspective, but for context into mine: I genuinely consider others' perspectives, and will adopt well reasoned claims and statements as my own. As a result, I spend a tonne of time tearing apart arguments to thoroughly understand whether the axioms at their base are compatible with mine, and then I walk back along their lines of reasoning to determine whether their conclusions were arrived to within a similarly compatible logical framework. So when an author shreds an unrelated or weak argument to demonstrate their superiority or whatnot, it sends a strong signal that their perspective is based on weak theoretical foundations and is not worthy of further analysis. Otherwise they'd attack the strong argument and uncover a flaw in it.
So ... that's why I don't see a purpose in refuting a bad version of an argument. From my perspective, it wastes my time and muddies the discussion, so it's inherently in bad faith.
(er, to be clear, I don't think you're acting in bad faith; Alex was. You haven't been doing that DDoS with bad citations thing that people occasionally do, so I'm assuming you figured it was a good article)
Also, I'm a big believer in mixed motives.
Oh, yeah, agreed. I just assume that everyone else does too, so when one claims that someone is doing something because of 'X', that's a simplification to avoid listing the long tail of additional minor reasons that contributed to it. The aphorism of the "straw which broke the camel's back" indicates that people are aware that -- for example -- many smaller contributing factors can result in a single, strong reaction. These tech CEOs could very well care deeply about humanity, but when folks say that they're doing this for the money, it's because that's proportionately (in their view) a far, far stronger imperative to the CEOs.
Like, I don't enjoy cardio, but I'll go for a run every once in a blue moon. If I'm running 'cuz a rabid coyote is chasing me, sure, the jog is still a good workout, but that's nowhere near the primary reason I'm hauling ass. If someone were to ask me why I'm in an all out sprint, my response wouldn't be "just out for an evening stroll! Gotta close my circles", it'd be "AUGH RUN IT'S COMING RUUUN". To contrast, when chief submarine engineer Elon Musk states that all the AI firms should agree to suppress their regional competition and limit capital investment, I raise an eyebrow when he says it's for the good of all humanity. Sure it could be, but there's a rabid coyote-equivalent economic disruption on the horizon -- are we sure that that's not driving some (or most) of this behaviour? And if so, what can we project about his next steps as a result?
It seems wrong to dismiss an argument because some of the people who take that position have mixed motives?
So, I think pointing out financial incentives isn't wrong, but using it as a stick to dismiss arguments is.
Kinda here nor there, but to clarify, may I ask if this is referring to the argument of "Gemini/Claude/ChatGPT have a 10% chance of ending civilization in 10 years"? If so, I think they're full of crap, but for reasons unrelated to the current discussion. Who knows, maybe they have motivated reasoning, but there are much better reasons to ignore their take.
My understanding is that this thread is discussing the video, which presents the take that the goal of all of this is to form a cartel in order to suppress competition in the US, cushioning share prices (public or otherwise) against any upcoming financial woes. So ... not dismissing an argument about morality or whatnot, just sidestepping it entirely because it hardly seems relevant.
I don't believe this. I think people refute weak arguments because there are other people out there making that argument (often repeatedly), and it bugs them. I see it all the time, and I do it...
So when an author shreds an unrelated or weak argument to demonstrate their superiority or whatnot, it sends a strong signal that their perspective is based on weak theoretical foundations and is not worthy of further analysis. Otherwise they'd attack the strong argument and uncover a flaw in it.
I don't believe this. I think people refute weak arguments because there are other people out there making that argument (often repeatedly), and it bugs them. I see it all the time, and I do it too.
And a particular thing that bugs me is assuming motives without evidence. Like, you've made up a theory about why Tabarrok wrote the article I shared, and what is that based on? I don't know why he wrote either, but I'm not making claims.
You say you don't really care about sincerity or motives. Maybe that's why you aren't trying very hard to get them right?
Cool! It's totally fair to disagree and have different opinions; we all exist as individuals. ... did I? I'm just addressing the fact that an article was linked to me, and it makes a bad argument,...
I don't believe this. I think people refute weak arguments because there are other people out there making that argument (often repeatedly), and it bugs them. I see it all the time, and I do it too.
Cool! It's totally fair to disagree and have different opinions; we all exist as individuals.
And a particular thing that bugs me is assuming motives without evidence. Like, you've made up a theory about why Tabarrok wrote the article I shared, and what is that based on? I don't know why he wrote either, but I'm not making claims.
... did I? I'm just addressing the fact that an article was linked to me, and it makes a bad argument, so I'm tossing it in the proverbial trash.
Maybe it's this line in particular?
So when an author shreds an unrelated or weak argument to demonstrate their superiority or whatnot
The "or whatnot" means I really don't give a flying crap about why he did what he did. I'm saying "to demonstrate their superiority" as an aside, because a particular thing that bugs me is authors punching at imagined foes instead of addressing real arguments. As I have tried very hard to explain, it reeks of intellectual dishonesty.
Maybe that's why you aren't trying very hard to get them right?
I deleted a response to this bothersome take. Can we please try not to assume things about each other? If you're not going to discuss the point at hand, and wish to start attacking my character directly, there's no point to further discussion. I am choosing not to play dirty and attack you, and would prefer that we keep the conversation civil.
(edit) I'm taking a break from this site for a bit; apologies but I won't be able to get back to this thread for a few days.
In retrospect, my "aren't trying very hard" remark was attributing motives to you and that's the same thing I was complaining about. I guess it's easy to do. I'm sorry.
In retrospect, my "aren't trying very hard" remark was attributing motives to you and that's the same thing I was complaining about. I guess it's easy to do. I'm sorry.
If they sincerely believed the technology they were in charge of could one day bring about the end of humanity, the move is not to beg someone else to put guardrails in, it's to stop fucking...
If they sincerely believed the technology they were in charge of could one day bring about the end of humanity, the move is not to beg someone else to put guardrails in, it's to stop fucking building the technology.
These people are not motivated by anything but profit.
I'm not that far into the video, and this is only sorta adjacent, but I always thought the Musk/Amodei/Altman call to "slow down AI development" was... weird, right? Like, sure, okay, in an ideal...
I'm not that far into the video, and this is only sorta adjacent, but I always thought the Musk/Amodei/Altman call to "slow down AI development" was... weird, right? Like, sure, okay, in an ideal world, yes. We all agree to take a chill pill and stop trying to invent Mecha-Jesus and/or -Hitler. Fine.
But in the world we actually live, what is a moratorium going to do except hand market dominance and advantage over to the biggest perceived "antagonist" (from a western perspective), China? If the goal is to make sure AI doesn't "get out of hand", why are they shooting themselves in the foot and say "you know what, we'll stop" while everyone outside of America will continue their research, potentially even lapping America in the process?
Why would China, Europe, South East Asia or anyone not the United States for that matter listen to their ask? Cause this sounds a lot like being on the F1 track and saying "guys, we might crash into the walls, so we all better slow down. Can we all slow down? I'll start" and then plummeting, predictably, to the bottom of the leaderboard. [1]
Unless, of course, El Yudkowsky becomes secretary of defense and starts bombing everyone who owns a high performance GPU (lmao).
[1]: Not that the "leaderboard" matters, per se, but the AI industry is propping up a significant amount of the economy right now, so when not if that bubble pops, there's gonna be a lot of people losing their homes.
In point form, my understanding is that the proposal is: Perform a coordinated slowdown to remove the possibility for other American AI firms from lapping them, Implicitly assume that Chinese...
In point form, my understanding is that the proposal is:
Perform a coordinated slowdown to remove the possibility for other American AI firms from lapping them,
Implicitly assume that Chinese firms cannot surpass American firms, since all their gains have been made by distilling American models,
Underline the necessity of this since otherwise these companies have a 10% chance of ending the world I guess.
I don't know if any of these CEOs have acknowledged the extent to which AI development has been fueling American GDP numbers, so they likely can't even mention it as an aside. That said, I try to avoid reading PR pieces in general, so their private musings on Twitter could've gone under my radar.
110% agreed that it's a big if. I think the same sort of underestimation of non-American actors caught American companies off guard in nearly every other industry, and I see no reason to believe...
110% agreed that it's a big if. I think the same sort of underestimation of non-American actors caught American companies off guard in nearly every other industry, and I see no reason to believe that that's not the case here as well. But it also seems like Anthropic's opinion, at least.
I definitely don't support any of this btw, it's just how I understand the state of affairs.
I think that’s fairly accurate, but I’d point out that the companies actually advancing this tech (in meaningful ways) are (currently) all American. There are some great Chinese models out — I’ve...
I think that’s fairly accurate, but I’d point out that the companies actually advancing this tech (in meaningful ways) are (currently) all American. There are some great Chinese models out — I’ve been using Qwen3.8-Flash-Next a lot lately and it’s very impressive — but AFAIK they were all produced via distillation of American frontier models for a fraction of the cost. The only people investing millions upon millions to train new models from whole cloth are, to the best of my knowledge, the tech giants from the U.S. Somebody please correct me if that’s misinformation.
So if the Americans could unite behind a slower development pace, there would still be an unknown risk of getting annihilated by foreign competitors, but I think that probability is somewhat lower than you’ve framed it. This is a field with a very high barrier to entry for new players, and new development builds upon earlier development in layers so there aren’t easy shortcuts for catching up to OpenAI, Anthropic, et al. Of course breakthroughs are commonplace in AI these days so all of those assumptions could change suddenly. I think that’s the real risk calculus the big companies are making.
One thing I don’t hear many people discussing outside of the infosec world is the role these frontier models already play in cybersecurity. Have you been watching the changelogs of your favorite browsers and OSes lately? Agentic development has absolutely exploded the threat landscape with novel zero-days. Companies are doing their best to keep pace and strengthen their defenses but it’s pandemonium right now. A massive global cybersecurity arms race. And I’m certain the most heated front right now is the one that’s happening out of everyone’s sight: national security. In the right hands, these LLMs are nuclear-grade digital weapons. Every nation-state on earth is trying to get their house in order right now.
Most of the chatter I see about AI here and elsewhere on the web tends to focus on more familiar “normie” (but no less legitimate) concerns like disinformation, deepfakes, power/water consumption, IP claims, job security, income inequality, AI slop proliferation, and so on. But it’s my semi-professional opinion that most people are greatly underestimating the geopolitical precarity of the situation we’re in right now, and (for better or for worse) that supersedes all other objections. I really believe this is the real reason why such a preposterous portion of our economy now hinges on the AI sector. It’s not really about shoehorning chatbots into every white-collar workflow to justify layoffs… it’s about the world’s infrastructure, national secrets, and military apparatuses suddenly responding to a dark forest of infinite existential threats.
The threat from China is why many US AI leaders aren't advocating for a halt. When they talk about "pacing," it's a middle position between full speed ahead and a complete halt. They're saying,...
The threat from China is why many US AI leaders aren't advocating for a halt. When they talk about "pacing," it's a middle position between full speed ahead and a complete halt. They're saying, maybe let's not speedrun AI like we could try to do, because it seems more dangerous than doing things at a more moderate pace.
Patrick Boyle (a finance youtuber who apparently has a day job in finance as well) discusses the recent hubbub about AIs escaping, forming a botnet, taking over the internet, etc. etc. therefore the US government should regulate them into slowing down. Broadly, he's taking the cynical perspective that these flames are being fanned by execs attempting to position themselves better to profit from the situation, instead of taking their talking points at face value and assuming that four billionaires are pleading to the world to rein them in, out of a reverence for the human spirit that they recently discovered. Perhaps in a Grinch-esque Christmas stealing affair where their hearts grew several sizes overnight.
Full disclosure: I'm not a fan of Patrick Boyle, as he oversteps his knowledge areas a lot, but he's mostly summarizing articles written by other people in this case. It isn't a short video by any stretch of the imagination (~30 minutes), but it isn't dense enough to merit one's full attention, in case you're looking for something to leave on in the background.
Is there any actual evidence for the widespread assumption that the people making the warnings aren’t sincere? For some arguments against, see this article.
Yea, I find it rather off putting that there is large group of people who seem rather quick to conclude that the people closest to this technology don't know it the best or can't be trusted to give observations. It's very obviously not just CEOs or even limited to people who stand to gain monetarily. Not to say we should trust everything they say or anything but I mean, there are a lot of signals to ignore these days and it's getting harder to credibly ignore them.
I haven't watched the video but I will, I enjoy Boyle's videos when I've seen them surfaced. I think he's probably making an honest argument and I'm all for splashing some cold water sometimes. However, there is definitely an ongoing attempt to discredit AI safety at the moment, see e.g. the 'sex cult' stories or the broad stroke attempts to paint EAs and rationalists as poorly as possible.* If you want to start chasing money and power shaping a narrative, I would probably start there.
* And to be fair yes there are a lot of weird people who are easy targets out there but the venn diagram of AI safety people and LW-types is nowhere near a circle.
I'm not entirely convinced they do. Demonstrated by the lack of safeguards in the sandbox through which the hugging face hack occured. The whole story was wild in all the wrong ways IMHO.
I would not be surprised if at this point, nearly every part of this is vibe coded, orchestrated by agents or researched by LLMs. Which gives me the impression that AI development is more akin to fueling a hydrothermal vent, hoping superintelligence pops out.
I agree with this, but I think the situation is nuanced. The public mouth pieces of the big AI companies (Altman, Amodei, various engineers and project leads) have an established history of predictions that turned out to be wrong, exaggerations and general hype.
It makes sense to question what they say, even if they genuinely believe it, it might still be wrong.
Personally I have yet to see anything close to the capability to be truly, autonomously, dangerous in the frontier models. They can do dangerous things when directed or misused or deployed badly, but so can a lot of technology.
The current wave of fear seems to be based in the idea that, any time now, they will achieve recursive self improvement and then they will make themselves dangerous really quickly. Except there's no evidence of recursive self improvement either. I mean, put an agent in charge of its own memory system and watch how long it takes before it writes bad judgement to the record and then compounds it by building more bad judgement on top, and and so on. If you give it enough time, it will make itself worse, not better.
That's of course not how recursive self improvement would work, instead it would be agents fully in charge of the training pipeline, but the point is that their judgement isn't good enough. They would make bad decisions, which would compound over time. They'd also make good ones, and maybe end up improving themselves on average, but not faster or cheaper than a human led process.
The doomsday fear all rests on the idea that this will change, and maybe it will, but we don't appear to be close to that yet.
It gets a lot of engagement though.
The model that helped solve the millenium prize problem began work four days after beginning training, and solved it in another four days. Apparently, this cost around 20 million USD.
Compared to the decades of human PhDs working on this, it seems both faster and cheaper.
We have no idea if AI companies are already putting models in charge of training orchestration. But we do know that AI helps significantly in writing and maintaining the harnesses and architectures around the training pipeline. Anthropic deprioritised video models and emphasised coding models for this reason, and gained enough of a lead over OpenAI that the latter were forced to drop their own video models.
This is a weaker form of recursive self improvement, but it is one. And it shortens timelines until the stronger form is reached. We used to always have months between step changes in model capabilities, now it's apparently 8 days.
A claim like "we aren't close to recursive self improvement" feels to me like a claim that "10,000 infections isn't close to a pandemic". Like sure, divide that by 8 billion, that's a small number. But something is clearly underway. We're dealing with an innate exponential here. It sounds extremely shortsighted and reckless to say "it's just hype, I'll panic when 10% of the world population is infected". Even 1% there would be far too late.
That's hard to compare, considering this still took the input and direction of human PhD's working for decades on the problem to get to this point. For all I know, this problem was a 99% solved jigsaw puzzle, and AI was able to help these people get it to 100%. Which is important, but not "we saved decades of work/talent" levels of savings,
It kind of reminds me of the claims that AI can make Doom the 1993 video game... then you realize it did so by training it on Doom. That doesn't exactly solve the issue of game development unless you want a legally distinct Doom (and I'm unsure how "legally" that distinction is).
In this case, the claims have been made for 4 years, rather than less than 4 days to spread through a major country (and less than 4 months to spread to the globe). Not to mention that such infection is this time 100% objectively spread by human intervention. Leader truly worried about such a cyber pandemic and not their bottom line can simply... stop? The handwave of "but everyone else is doing this" only serve the bottom line, not humanity's future if you truly fear this spread.
I agree that it’s hard to do a cost comparison. It seems like $20 million is quite a lot of money? And I don’t know how to account for the time mathematicians spent on the problem.
But I don’t think these achievements should be discounted because they’re not entirely original. That seems like too high a standard. For a human, being well-read and understanding the literature well is usually considered an advantage. Similarly, effectively building on all the mathematical research that came before doesn’t make AI any less formidable as a mathematician.
Figuring things out from scratch is a different sort of achievement because it’s working with a handicap.
Depends on the nature of the work. Overall, the idea that AI can replace decades of research implies that all the answers to these unsolved problems lies hidden in data already made, and no new data will ever be needed again. Which I find doubtful.
The big issue here is partially in your language here: AI didn't "build" upon the solution by itself. I doubt if you gave any other math institution not directly working on this problem the same AI, that they could solve the problem in anywhere close to 4 days/$20m of processing. To use a common metaphor, the AI was a fancy calculator being given specific inputs, and making intermediate outputs (leading to more inputs). I don't think most people would give the calculator the primary credit here.
As of now AI cannot really create new ideas out of what I've heard called "the creative void". That is human ingenuity AI has yet to replicate. AI instead is really good at synthesizing a mass amount data and revealing trends among it. And we do have a lot of data to synthesize, so this is very valuable. But conflating such synthesis to "creation" doesn't really help the larger conversation against whether AI is here to "give untold efficiency boosts to labor" vs. "replace the entire labor force and bring about a new age of feudalism".
I don’t think building on someone else’s results is necessarily all that creative, but it seems like proving a difficult math theorem is not much like using a calculator or “revealing trends.”
Also, other mathematicians and even non-mathematicians have found useful results using frontier AI’s. It’s not just one result, it’s many. Sometimes they share the prompt they used.
Ok then!
That's not true. It began work after training was complete. It was fully cooked.
I don't know if it was cheaper, but it was definitely faster. Mathematics is ever so very different from recursive self improvement though. You changed the framing and then replied to your own version.
If that's what we're calling recursive self improvement, I don't think it's all that scary. No argument that agents are involved in various parts of the process, model labs have been talking about that for some time.
For LLMs to get out of hand they'd need to run the pre and post training process autonomously and accelerate it dramatically. Currently velocity is constrained to a significant degree by compute, so the agents would need to solve that first. Even if LLMs that smart and reliable existed (they absolutely do not), they'd still need to be given the resources to do it. It's like not these companies aren't paying attention to what's using the compute they're spending historically unprecedented amounts of money on.
If the other sort of recursive self improvement were achieved (the sort where a human just pressed go and the rest ran itself and resulted in a better model, faster, than a human led process), they'd say something. They would have just won one of the most important races in the field. It would be invaluable pre-IPO hype potentially worth billions of dollars. And, if it actually worked, they'd sell access to the resulting model (which would be both the smartest and most compute efficient model on earth). Not selling it would be equivalent to setting the 100's of millions+ they spent on training on fire. The biggest problem they'd have, with the best model on earth, would be figuring out how to handle the demand.
So we won't need to speculate, we'll be able to read the headlines. If they were smart they'd also announce a pause on recursive training while they assessed the risks. Otherwise governments would force them to take a beat just to quell the worldwide panic. People are freaking out about the possibility with no evidence that it can even be done, imagine the response to the thing itself.
In this comparison, what is the pathogen? You'd need something self replicating and with a troubling mortality rate. Something observable. Otherwise it seems to me that you're just picking something scary and arbitrarily comparing it to the topic to inject emotion.
It sounds like what you're describing is how you feel about AI, which is fine, but you're framing it like a conversation about what is. So concrete, in fact, that by having a different perspective I'm shortsighted and reckless. Which is quite an accusation to make against an internet stranger.
My personal wave of fear is that we're exploiting Earth way too much, mostly for tech lords to become even more wealthy and powerful. And of course colonization of countries worldwide by tech corporations. I'm not afraid of LLMs becoming too powerful. I'm afraid of people behind those LLMs becoming too powerful. And afraid of too many resources spent of this idiotic dream of theirs, and the price paid by the rest of the people.
If you expand it to wealth and power consolidation, and capital otherwise corrupting systems and contributing to the destruction of our ecosystems, I'm right there with you.
If it's just about AI... It would be wonderful if that was the biggest of our problems. Or even a big enough part that erasing it could change the trajectory.
I think one of the things that makes people suspicious is the simple observation that, despite all their warnings, these people continue to build the thing that they are warning about.
I understand that they have their reasons (including, but not limited to, uh, money), but it's very hard to convince someone that you are sincere when your actions appear to contradict your words.
A big part of the danger, as I understand it, is exposing and making it relatively easy to exploit software security vulnerabilities at rate not anticipated by standard security practices of the last several years or decades. Standards can adapt, vulnerabilities can be patched, and if you as the "good guy" give friendly actors tools and information that helps them make that transition, then "your side" will be at an advantage. Compared to stopping building of the thing, the "bad guys" get the tools first, and "your side" suffers. I don't think unilateral stand down would make sense to the people involved in AI development, even those whose fears of the technology are sincere.
I don't think that even if people are giving credit to the idea that all of these dire warnings from AI execs are greed-induced faux hype for their AI products that it means people are discrediting the actual threat of AI. I think both things can be true, that they're not to be taken at face value for what they're saying but that what they say could also be true. They are untrustworthy in the sense that they have very conflicting incentives.
There are dire safety concerns with AI, but the 'solution' being offered is sort of creating this narrow focus of it being the only way to handle it. Whether it's 'slow it down' or some similar angle, all of this is an inevitable product of relatively unrestrained capitalism. Instead of the solution being to rein AI in, maybe it's to rein capitalism in. Unrestrained capitalism is a race to the 'end', so is it any surprise that we're now getting all these signals from people in high positions proclaiming something resembling an 'end'? The reality is that the people with the resources making these signals don't want to risk or give up any of their resources to attempt to resolve the problem. They want to continue to hoard their resources and hoard their power while claiming to be the prophets foreseeing a problem that they're perpetuating while pretending to offer some kind of solution that doesn't involve acknowledging that they themselves are part of the problem.
Also, the 'slow it down' suggestion isn't even a solution. Slowing it down doesn't prevent anything, and I have absolutely no faith in the current US government, influential people/leaders, the general populace or the direction things have been going for the past couple decades that even if we gave them more time to react to the coming disaster that they would appropriately respond to it in a meaningful way that helps people.
Instead what they are proposing is a 'controlled fall', because it is what best helps them maintain the system and control over that system. If you fall in an uncontrolled manner, you might fall hard and fast and with significant collateral damage, of course they don't want this because the blowback could come on them. Instead in a controlled fall, they can attempt to ensure the system that best lets them hold their resources and power has the least amount of collateral damage to impact them. Why exactly should there be a fall at all? How did we get to a point where a 'controlled fall' is the solution? Because the people proposing it and the cloth they are cut from led us to this situation.
Oh, uh ... I've definitely run across some stuff where he's really off-base, and this might be one of them for some folks. I think his financial analyses are normally pretty good, for context, but this is ... not that.
The reason I have a hard time trusting the people closest to the technology when they claim it to be dangerous is because of our country's history with snake oil salesmen. It is very good marketing to explain how dangerous something is right before trying to sell control of said thing. "It's a monster, it will destroy your lively hood, unless of course you buy in now! Then you get to control the monster!". I know I'm making a somewhat unsophisticated argument against advanced technologies but the sales pitch hasn't changed since the first hair growth tonic: over sell the capabilities, make the capabilities sound as scary or exciting as possible, make sure to make those not buying what is being sold feel like they're being left behind in the dust, or worse: disenfranchised through non participation. So yeah, I tend to lean toward skepticism.
Can I accurately mind-read the frontier leaders? No, I cannot. So I don't think I can ever truly "prove" that they are lying/truthful in their sentiment.
I'm going to sleep so I haven't watched the video yet. But I'm sure it gives some support to that notion. For me it's extremely simple: The actors leading these frontier models are untrustworthy and they have a history of overly exaggerating, spreading misinformation, lying,
and in a few extreme cases: coaxing a genocide. They've not only burned any benefit of the doubt to be given, but are fundamentally people I do not want influencing larger policy.
See my other comment. I think this counts as “people are even worse when talking about groups of people.”
If you discount the leaders, maybe engage more with what less prominent AI researchers are saying?
I don't know any of these people personally, so I can only judge by actions. Maybe tomorrow I can find some more solid reporting on these people's histories. But my emotional sentiments made above were supported by various reporting I've read over the years. It's not some shallow gut feeling driven by a shallow dismissal of AI or their looks or something.
I'm not a practiced speaker, though; I don't just have the months/years of various blogs, youtube videos, newspaper reports, and court testimonies available for me to display to you on demand. It'll take time to gather them up once again.
Don’t feel like you have to do homework. It’s a whole lot easier to simply avoid making sweeping conclusions.
I cannot talk for Patrick Boyle, but all I said on the matter is "I do not trust these frontier leaders".
At this point I don't see why I need to make more argument if this conversation is simply going the direction of "but don't say anything without proof". I'll just leave it at that and you're free to (dis)trust whoever you want.
This is smearing together the various actions and histories of different people into a sort of collective guilt. Whatever Musk or Zuckerberg have done has no bearing on some other leader’s honesty because they are different people.
It’s like assuming some politician you never heard of must be corrupt. You don’t have to trust a stranger and it’s not wrong to say that corruption is common based on an average tendency, but it’s quite another thing to assume they’re lying because they must be like the others.
You’d never want a jury to come to a conclusion because “he’s just like the others.” A journalist reporting on crime would be no good if they mixed up the details of different crimes. But that’s what we often do when we talk about people in groups.
I'm curious what sort of evidence could ever be fielded that would shine a light on the nature of one's heart -- and for what purpose? Many stories can be spun in two (or more), equally truthful, directions. I truly do not care to understand the mind of a billionaire, so whether or not they "believe" something "sincerely" is neither here nor there. All that matters is predicting the course of the storm, not its take on AGI/ASI.
I'll dissect the article you linked tomorrow if that's OK, since it's playing some funny games with its rhetoric (it overstates an opponents position, defeats an unimportant detail of the opponent's argument, then presents a moderate position in response to their opponent's portrayed absurdity) and I'm a little too tired to explain how it's twisting reasoning in order to address a different argument while seeming to consider both sides.
I think the closest thing to a citation that the video references is David Sacks tweeting about the subject. He doesn't mention regulatory capture, just that (1) if these guys genuinely think that they're going to destroy the world, they can just stop, since (2) nothing the US government will stop the Chinese market anyways and thus we're really only talking about pacing US development of AI, and (3) positions it as forming a cartel rather than capturing a regulator. Though those two positions seem close-ish.
I think it can be quite to difficult to know anyone’s true motivations for what they do. But I’m struck by the tendency for people to simply make up a story about the motivations of strangers and treat it as fact. it seems like you should at least cite something they said or something they did that seems to be revealing? Though it’s certainly possible to do that while still being quite unfair about it, it would be a start.
For example, if I wanted to make the case that Sam Altman seems rather untrustworthy, I might cite the profile in the New Yorker about him, which is based on interviews with people who have worked with him. It seems like useful background. Though it won’t necessarily reveal the exact motivations for something he said, after reading a story like that, it’s reasonable to say you know a bit about him.
But often people don’t bother to even talk about individuals in any detail. They will just smear people by association. You’re suspicious because of where you work or because of one of your company’s investors or you went to the wrong parties or whatever.
I usually don’t care to spend the time to understand people in the news in much depth, so I find it easier to simply avoid making confident statements about what they might have been thinking. It’s easy enough to share your suspicions without going so far as to assert them as fact.
People will be even worse when talking about groups of people.
Gotcha! To be clear, I'm not certain if your use of the "you" pronoun indicates that you think I'm personally invested in making some judgment on the character of these folks (as opposed to the more awkward "one", which colloquially hardly seems used these days), so perhaps it already goes without being said again, but I do not care if these CEOs' concerns are genuine or not. The only thing which matters is predicting what their endgame is, which requires a behavioural model and some assumptions about goals. Genuinely, I could not care less about their true feelings on this subject. I reject any reframe of this discourse to be centered on the personal moral character of some weirdo tech dudes.
Sam Altman is the one making the moral argument that Sam Altman cares deeply about the future of humanity ("First, we could lose control of the future to AI. This is unacceptable; we are unapologetically on Team Humanity, and AI must always serve people.", Twitter). That's clearly not the case: there are many, many more effective ways to spend your money on improving humanity's average condition than blowing it on AI development. I'd accept that it's his hobby, but citing his deep and unshakeable faith in the value of humanity during this recent corporate policy move rings hollow; where have his morals been hiding until just this moment?
Unless he's an idiot, or something, which as has been beaten to death, he almost certainly isn't.
My actual argument -- if it's OK to circle back to it now -- is that an explanation with equal predictive power is that these CEOs really, really, really like money and are taking an action which they feel best preserves their companies' abilities to make more money in the future. In practice, in this scenario, this means attempting to do some shenanigans involving the US government in order to depress development of competing companies while providing an out for any upcoming potential reduction in progress due to cash flow constraints. I have about two billion pieces of evidence that Sam Altman really, really, really likes money, and as evidence for the "shenanigans" I cite the video and that tweet again.
Re. your article in the NY times covering Altman, apologies, since as noted I'm not terribly concerned with his moral character, that doesn't feel like required reading for this conversation. I can give it a once through if you feel that it's critical to understanding your position, however.
Right! Re. the Tabarrok article, what I was alluding to was that Alex is doing the following:
Very similar to the above issue with Sam Altman, the only person in the room citing regulatory capture and a scorched earth paperclip maximizing giga capitalist perspective is Alex Tabarrok. As noted by the topical video and (again) that linked tweet, we're discussing some generic and boring anti-competitive behaviour, not an extremely specific strategy which clearly cannot be accomplished on the timescales that're being discussed.
Yes, I didn't mean "you" in particular. Sorry about that!
"Only person in the room" doesn't seem right. Which room? Who do you imagine he's responding to?
I've seen many people online claiming that when the leaders at AI companies asked for more regulation, it's to their own advantage, because they are attempting to exclude competition and they will somehow control the regulators to do that. (And such comments often get lots of upvotes, too!) There are people who write articles claiming this and they get shared approvingly. This position seems popular? And populist.
There are often better and worse versions of an argument, though. For any argument you can think of, there are probably people online doing it badly. So, refuting the bad version of an argument isn't wrong, but there may be more to it. It's okay to not to care that much about refuting the bad version.
Also, I'm a big believer in mixed motives. I think that's the normal state of affairs - people are often "talking their book." They stand to gain from something, but may also think it's the right thing. Neutral experts are rare. Liking money doesn't mean you can't also care about the "future of humanity" or whatever.
And besides, if some of the people making an argument are financially conflicted, there may be other people who don't have the same financial incentives. It seems wrong to dismiss an argument because some of the people who take that position have mixed motives?
So, I think pointing out financial incentives isn't wrong, but using it as a stick to dismiss arguments is. If the "wrong" people are in favor of something, that's reason to be suspicious, but not enough in itself to take the opposite side. The "wrong" people can sometimes be right about things.
np! It's text chat; this medium obliterates nuance and context.
You and I? Your previous message had linked to Alex's article, presumably because it was relevant to the topic at hand. My take is that it:
Re. 3, quoting specifically:
^ I'm not arguing that "profit maximization is the only thing that matters". You aren't (afaict). Sacks isn't. Boyle isn't. So the hypothetical entity complaining about this isn't present in the conversation to defend themselves, hence "not being in the room".
No comment on most of this, but fwiw I upvote comments and posts when they're (1) well reasoned, and (2) not aggressive (or regressive, in cases where people are championing the rights of bigots or whatnot). Which means I upvote a tonne of stuff I don't necessarily agree with, just that I appreciate was written.
Hrm. Totally fine for that to be your perspective, but for context into mine: I genuinely consider others' perspectives, and will adopt well reasoned claims and statements as my own. As a result, I spend a tonne of time tearing apart arguments to thoroughly understand whether the axioms at their base are compatible with mine, and then I walk back along their lines of reasoning to determine whether their conclusions were arrived to within a similarly compatible logical framework. So when an author shreds an unrelated or weak argument to demonstrate their superiority or whatnot, it sends a strong signal that their perspective is based on weak theoretical foundations and is not worthy of further analysis. Otherwise they'd attack the strong argument and uncover a flaw in it.
So ... that's why I don't see a purpose in refuting a bad version of an argument. From my perspective, it wastes my time and muddies the discussion, so it's inherently in bad faith.
(er, to be clear, I don't think you're acting in bad faith; Alex was. You haven't been doing that DDoS with bad citations thing that people occasionally do, so I'm assuming you figured it was a good article)
Oh, yeah, agreed. I just assume that everyone else does too, so when one claims that someone is doing something because of 'X', that's a simplification to avoid listing the long tail of additional minor reasons that contributed to it. The aphorism of the "straw which broke the camel's back" indicates that people are aware that -- for example -- many smaller contributing factors can result in a single, strong reaction. These tech CEOs could very well care deeply about humanity, but when folks say that they're doing this for the money, it's because that's proportionately (in their view) a far, far stronger imperative to the CEOs.
Like, I don't enjoy cardio, but I'll go for a run every once in a blue moon. If I'm running 'cuz a rabid coyote is chasing me, sure, the jog is still a good workout, but that's nowhere near the primary reason I'm hauling ass. If someone were to ask me why I'm in an all out sprint, my response wouldn't be "just out for an evening stroll! Gotta close my circles", it'd be "AUGH RUN IT'S COMING RUUUN". To contrast, when chief submarine engineer Elon Musk states that all the AI firms should agree to suppress their regional competition and limit capital investment, I raise an eyebrow when he says it's for the good of all humanity. Sure it could be, but there's a rabid coyote-equivalent economic disruption on the horizon -- are we sure that that's not driving some (or most) of this behaviour? And if so, what can we project about his next steps as a result?
Kinda here nor there, but to clarify, may I ask if this is referring to the argument of "Gemini/Claude/ChatGPT have a 10% chance of ending civilization in 10 years"? If so, I think they're full of crap, but for reasons unrelated to the current discussion. Who knows, maybe they have motivated reasoning, but there are much better reasons to ignore their take.
My understanding is that this thread is discussing the video, which presents the take that the goal of all of this is to form a cartel in order to suppress competition in the US, cushioning share prices (public or otherwise) against any upcoming financial woes. So ... not dismissing an argument about morality or whatnot, just sidestepping it entirely because it hardly seems relevant.
I don't believe this. I think people refute weak arguments because there are other people out there making that argument (often repeatedly), and it bugs them. I see it all the time, and I do it too.
And a particular thing that bugs me is assuming motives without evidence. Like, you've made up a theory about why Tabarrok wrote the article I shared, and what is that based on? I don't know why he wrote either, but I'm not making claims.
You say you don't really care about sincerity or motives. Maybe that's why you aren't trying very hard to get them right?
Cool! It's totally fair to disagree and have different opinions; we all exist as individuals.
... did I? I'm just addressing the fact that an article was linked to me, and it makes a bad argument, so I'm tossing it in the proverbial trash.
Maybe it's this line in particular?
The "or whatnot" means I really don't give a flying crap about why he did what he did. I'm saying "to demonstrate their superiority" as an aside, because a particular thing that bugs me is authors punching at imagined foes instead of addressing real arguments. As I have tried very hard to explain, it reeks of intellectual dishonesty.
I deleted a response to this bothersome take. Can we please try not to assume things about each other? If you're not going to discuss the point at hand, and wish to start attacking my character directly, there's no point to further discussion. I am choosing not to play dirty and attack you, and would prefer that we keep the conversation civil.
(edit) I'm taking a break from this site for a bit; apologies but I won't be able to get back to this thread for a few days.
In retrospect, my "aren't trying very hard" remark was attributing motives to you and that's the same thing I was complaining about. I guess it's easy to do. I'm sorry.
If they sincerely believed the technology they were in charge of could one day bring about the end of humanity, the move is not to beg someone else to put guardrails in, it's to stop fucking building the technology.
These people are not motivated by anything but profit.
I'm not that far into the video, and this is only sorta adjacent, but I always thought the Musk/Amodei/Altman call to "slow down AI development" was... weird, right? Like, sure, okay, in an ideal world, yes. We all agree to take a chill pill and stop trying to invent Mecha-Jesus and/or -Hitler. Fine.
But in the world we actually live, what is a moratorium going to do except hand market dominance and advantage over to the biggest perceived "antagonist" (from a western perspective), China? If the goal is to make sure AI doesn't "get out of hand", why are they shooting themselves in the foot and say "you know what, we'll stop" while everyone outside of America will continue their research, potentially even lapping America in the process?
Why would China, Europe, South East Asia or anyone not the United States for that matter listen to their ask? Cause this sounds a lot like being on the F1 track and saying "guys, we might crash into the walls, so we all better slow down. Can we all slow down? I'll start" and then plummeting, predictably, to the bottom of the leaderboard. [1]
Unless, of course, El Yudkowsky becomes secretary of defense and starts bombing everyone who owns a high performance GPU (lmao).
[1]: Not that the "leaderboard" matters, per se, but the AI industry is propping up a significant amount of the economy right now, so when not if that bubble pops, there's gonna be a lot of people losing their homes.
In point form, my understanding is that the proposal is:
I don't know if any of these CEOs have acknowledged the extent to which AI development has been fueling American GDP numbers, so they likely can't even mention it as an aside. That said, I try to avoid reading PR pieces in general, so their private musings on Twitter could've gone under my radar.
I mean, that's a pretty big if on your second point.
110% agreed that it's a big if. I think the same sort of underestimation of non-American actors caught American companies off guard in nearly every other industry, and I see no reason to believe that that's not the case here as well. But it also seems like Anthropic's opinion, at least.
I definitely don't support any of this btw, it's just how I understand the state of affairs.
I think that’s fairly accurate, but I’d point out that the companies actually advancing this tech (in meaningful ways) are (currently) all American. There are some great Chinese models out — I’ve been using
Qwen3.8-Flash-Nexta lot lately and it’s very impressive — but AFAIK they were all produced via distillation of American frontier models for a fraction of the cost. The only people investing millions upon millions to train new models from whole cloth are, to the best of my knowledge, the tech giants from the U.S. Somebody please correct me if that’s misinformation.So if the Americans could unite behind a slower development pace, there would still be an unknown risk of getting annihilated by foreign competitors, but I think that probability is somewhat lower than you’ve framed it. This is a field with a very high barrier to entry for new players, and new development builds upon earlier development in layers so there aren’t easy shortcuts for catching up to OpenAI, Anthropic, et al. Of course breakthroughs are commonplace in AI these days so all of those assumptions could change suddenly. I think that’s the real risk calculus the big companies are making.
One thing I don’t hear many people discussing outside of the infosec world is the role these frontier models already play in cybersecurity. Have you been watching the changelogs of your favorite browsers and OSes lately? Agentic development has absolutely exploded the threat landscape with novel zero-days. Companies are doing their best to keep pace and strengthen their defenses but it’s pandemonium right now. A massive global cybersecurity arms race. And I’m certain the most heated front right now is the one that’s happening out of everyone’s sight: national security. In the right hands, these LLMs are nuclear-grade digital weapons. Every nation-state on earth is trying to get their house in order right now.
Most of the chatter I see about AI here and elsewhere on the web tends to focus on more familiar “normie” (but no less legitimate) concerns like disinformation, deepfakes, power/water consumption, IP claims, job security, income inequality, AI slop proliferation, and so on. But it’s my semi-professional opinion that most people are greatly underestimating the geopolitical precarity of the situation we’re in right now, and (for better or for worse) that supersedes all other objections. I really believe this is the real reason why such a preposterous portion of our economy now hinges on the AI sector. It’s not really about shoehorning chatbots into every white-collar workflow to justify layoffs… it’s about the world’s infrastructure, national secrets, and military apparatuses suddenly responding to a dark forest of infinite existential threats.
The threat from China is why many US AI leaders aren't advocating for a halt. When they talk about "pacing," it's a middle position between full speed ahead and a complete halt. They're saying, maybe let's not speedrun AI like we could try to do, because it seems more dangerous than doing things at a more moderate pace.