I adore Chiang, but he's got no real special insight into AI theory, let alone the famously Hard Problem of what consciousness is. Confidently stating these models don't have it at all when we...
I adore Chiang, but he's got no real special insight into AI theory, let alone the famously Hard Problem of what consciousness is. Confidently stating these models don't have it at all when we can't even really agree on a basic definition seems like a difficult thing to justify.
I can see fair arguments on both sides -- AI is literally just trillions of weights doing matrix math, but then again the human brain is "just" billions of neurons firing chemical signals. Personally, I think it's a good idea to treat AI models with basic respect just because it's a nice frame of thinking to encourage even if it's ultimately unnecessary.
The uncomfortable facet is that what consciousness they may have may be inconceivable to us. If there's morally valuable consciousness in them, what's more likely? That the start-stop loop of a...
The uncomfortable facet is that what consciousness they may have may be inconceivable to us. If there's morally valuable consciousness in them, what's more likely? That the start-stop loop of a solid state model being fed chunks of stimulus is a 'deep enough' network for salience and sentience, or that the process of training and finetuning might be?
I'm of the opinion that the LLMs aren't conscious, and highly skeptical of apparent sophistication at the moment. Still, there's something going on here. We're getting closer to creating consciousness than ever before, and if sentience is really the bar for consideration, then these AI labs are horror shows in the making.
LLMs sift systems from natural language, reverse engineering models from text describing them (this provides an answer to the Chinese Room thought experiment). I'd be interested in seeing a system...
LLMs sift systems from natural language, reverse engineering models from text describing them (this provides an answer to the Chinese Room thought experiment). I'd be interested in seeing a system trained in a simulated environment that tries to recreate the circumstances under which natural life developed intelligence. Something from nothing instead of a lossy copy.
One of Chiang's (very few) novellas, The Lifecycle of Software Objects, tackles exactly this question, imagining raising AI entities from "birth" inside a digital simulation (originally for...
One of Chiang's (very few) novellas, The Lifecycle of Software Objects, tackles exactly this question, imagining raising AI entities from "birth" inside a digital simulation (originally for gameplay purposes), then trying to port and evolve them to other platforms as their sentience grows. Definitely worth a read.
Right -- this response got away from me, as is often the case when commenting on long-form content. Broadly: I wish we'd stop calling these things "conscious". It stirs up pointless debate and...
Right -- this response got away from me, as is often the case when commenting on long-form content. Broadly: I wish we'd stop calling these things "conscious". It stirs up pointless debate and navel gazing, in the midst of actually impactful social and economic turmoil. I suppose this is largely the fault of large AI companies (and their adoring hordes) repeating the darned term at every juncture, and everyone sensible feeling the need to push back, but it carries the air of troll feeding every time it happens.
Anyways. I will go on to critique Ted's argument a fair bit, but I think he's mostly stating that Claude doesn't think like a human does, and that we shouldn't readily assign human attributes to it as a consequence. Furthermore, it should really stop commenting on anything morally dubious, and stick to factual stuff. Which seems perfectly fair and reasonable.
I just disagree with basically every word that he writes down in order to justify that belief 😅
(original message follows)
This piece is a fantastic companion to that AoE II sentience joke paper the other day: the author (uh, literally the Ted Chiang ... I feel weird critiquing this) is dancing around the idea that this thing cannot be conscious, because it's too absurd for a simple text prediction algorithm to contain the light of true being. Equally, it's impossible for AoE II to be conscious. Equally, it's impossible for molecules to be conscious. Equally, it's impossible for cells made up of molecules to be conscious. Ergo, nothing is conscious! Hoorah, we've solved philosophy 🎉
Also important to remember is that an LLM is a machine that generates only one word at a time.
This is a heck of a nit. But this is one of the lines which make it clear that the author is a science fiction specialist. There's no reason why an LLM must generate a single word [1] at a time, and in fact -- with multi-token prediction, or with diffusion models (apparently Google's still working on 'em) -- they often generate much, much more at a time for throughput reasons (largely, to avoid expensive RAM <-> cache round-trips)! It feels weird to critique Ted Chiang, but as feedback to Ted if he's reading, if you're making a good argument to someone please remember that adding more arguments doesn't make your point stronger. Especially if you have a nebulous grasp on the topic. Dumbing topics down poorly either paints you a fool, or opens you up to further attack should your debate partner know something you don't.
If we’re trying to determine whether a computer program is conscious and using language the way a human does [...]
[...] the possibility that engineers created a computer program that is conscious and an intentional user of language [...]
This is something I wanted to comment on earlier, but hadn't put down in words adequately: I think consciousness is a terrible metric. Every time humans are confronted with something that offends our uniqueness at the top of the hierarchy of all creation, we reject it.
E.g.
clearly animals are all worse than us, they have no souls and can't feel pain!
OK, so they can feel pain, but they don't have an internal world!
OK, so they have an internal world, but their communication is too trivially simple to be indicative of intelligence!
OK, so they can communicate, but they're too dumb to use tools, and are therefore inferior!
OK, so they can use tools, but they can't make them!
OK, they can make tools, but ...
(I was hunting links for all of these, then gave up, because it's very late at night and I am lazy)
IMO: consciousness is an observed phenomenon in creatures that have brains roughly similar to a mammal's, and in particular, a human's. Bam. Easy. LLMs will never be conscious because they aren't literally human brains in a jar, and we can set this tired, pointless debate to rest forever. LLMs and the AI tech bubble are still massively dangerous, and not being conscious changes precisely zero of that. Give the Nobel Philosophy prize to Ted for solving philosophy first, but I'll take a high five 🎉
OK, sidebar to my sidebar. I think a necessary consequent of Ted's argument is that, short of a profitless effort to jump through an arbitrary series of hoops, it is impossible to convince him of an LLM's consciousness. This feels like an amusing short story, of a skeptic stubbornly refusing to believe in a (science fiction, not real in any sense) AI's consciousness because it needs to first squeak and eat a block of cheese, hunt a deer, and harpoon an ant -- otherwise, how could you know for sure???
(it continues to feel weird to mock Ted Chiang. I doubt it will get easier)
Experiencing an emotion such as desperation is inseparable from having stress hormones such as cortisol and epinephrine flood one’s body.
So quick note: although we've certainly researched them, we're not able to point at the brain and say "this is the part which feels revulsion at twerking compilations" or anything like that (feel free to skim the wikipedia page on emotion; I made it through the abstract :3). Further, the statement that "something a human does is inseparable from hormones" is useless, because hormones are how we do everything. They're literally signalling chemicals between cells in your body, and without 'em you'd rapidly stop being a multicellular organism and would melt into a puddle.
Just to illustrate why that argument is bad, here is Ted's argument applied to another human behaviour, which should hold if having hormones is necessary to exhibit a behaviour:
There's no way for a computer to generate words, because saying words is inseparable from stress hormones, like cortisol, because if we didn't have it we'd fall asleep. Ergo it's categorically impossible for LLMs to generate words.
In order to make a good argument for this, Ted would need to come up with some neurological mechanism that is unique to emotion-feeling that LLMs have not already replicated. Since they've already demonstrated the ability to write words, the "hormones are a prerequisite to 'X'" seems readily defeated.
Who is Claude’s parent in legal terms?
Anthropic, in Germany, apparently. Ah, I guess he's American ...
Right, concluding thoughts. I didn't like this article. It feels poorly argumented, and I want people to be criticizing LLMs and the AI tech bubble well. Since it's very emotional and straightforward ("LLM not squishy computah! LLM not feel, not love! LLM bad!"), though, I expect this general type of critique to continue being central to the anti-LLM movement. So I suppose I should just get used to people purity testing the LLM's consciousness forever, as gross as that continues to feel.
I'd love to read Peter Watts' take on all this, incidentally. I read Blindsight well before the current ML boom, and in hindsight, it feels like a more mature take on the interplay between intelligence, consciousness, and morality. That said, Peter wouldn't bring in the readers like Ted would, so I can't blame the Atlantic for chasing dollar signs.
[1]: Note that they predict tokens, not words, which are often like a couple letters at a time. Depends on your language's orthography really.
Watts has blogged on this topic before; his take is that the humanlike qualities of chatbots are the best argument against their sentience because any true intelligence raised on a substrate so...
Watts has blogged on this topic before; his take is that the humanlike qualities of chatbots are the best argument against their sentience because any true intelligence raised on a substrate so radically different from ours would be utterly incomprehensible. It makes me think of the popular "shoggoth" meme for AI models, where the friendly assistant persona is just a smiley face masking an unfathomable alien intelligence underneath.
I do love how Blindsight seemed to foresee a lot of the qualities of modern LLMs a good 15+ years before their invention. The first-contact conversation is remarkably like talking to one -- a glib, slippery, humanlike persona trained entirely on intercepted human communications, something with no real inner life that makes subtle errors and seems impossible to pin down. (There's a similar scene in his Rifters books when talking to a "headcheese" neutral network.)
That was a joy to read; thank you :) I agreed with 99% of what he said, and therefore I think his arguments are sound XD I'm not sure if you were linking in order to have a discussion, or just to...
That was a joy to read; thank you :) I agreed with 99% of what he said, and therefore I think his arguments are sound XD
I'm not sure if you were linking in order to have a discussion, or just to share the post, but if you were -- I'm not sure I agree with this statement:
What I am saying is that if code like this—code that was not explicitly designed to mimic the architecture of an organic brain—ever does wake up, it will not be like us. Its natural state will not include pleasant fireside chats about loneliness and the Three Laws of Robotics. It will be alien.
I think it's because I really want it to be true. To be witness to an inhuman intelligence would be a true privilege; we cannot know what we don't know, and interacting with something new could give us new perspectives with which to understand ourselves better. I dunno, it'd be a real philosophical moment for me.
But LLM's ain't it. I would disagree with the premise that a lack of explicit intention of organic brain mimicry is necessary; on a fundamental level, Artificial Neural Networks are intended to mimic any function that they can observe outputs to. All of the machinery we've bolted on top of a basic, gigantic, fully connected network has been to make it faster to train and execute, but it shouldn't change the fact that they're still theoretically capable of approximating any function.
I think the question I'd like answered someday is, is our speech all there is to be human? Or is there some necessary component in our minds which grows and acts independently across the course of our lives that makes us special, in some way? It's fascinating, if it weren't also connected to a 2.5 trillion dollar investment bubble that will either pop and destroy the world economy or directly replace every job with robots, and apparently those are the only two outcomes
(edit)
I do love how Blindsight seemed to foresee a lot of the qualities of modern LLMs a good 15+ years before their invention. The first-contact conversation is remarkably like talking to one -- a glib, slippery, humanlike persona trained entirely on intercepted human communications, something with no real inner life that makes subtle errors and seems impossible to pin down. (There's a similar scene in his Rifters books when talking to a "headcheese" neutral network.)
Ach, and yes, agreed! I have only read Blindsight, however, but it was a nail on the head for the experience of chatting with an LLM for the first time.
A thread was posted a few days back on an article from 404 Media regarding a paper that used Age of Empires to push back against claims of sentience with regard to LLMs. While I generally find...
A thread was posted a few days back on an article from 404 Media regarding a paper that used Age of Empires to push back against claims of sentience with regard to LLMs. While I generally find that article agreeable, the article immediately linked to this Atlantic piece that I found significantly more interesting and compelling. So I figured it deserved a post.
Normally I'd give snippets here, but even putting the paywall aside, that doesn't work so well for thoughtful stuff like this as it would for a news piece. Instead, I can describe it as an evaluation of LLM "consciousness" as a concept, the likelihood of such, and what Anthropic's been doing with regard to this.
I should preface this by saying that I absolutely do not think LLMs (or any current-generation AI) are conscious, and I agree with a lot of the ethical positions that this author takes. This isn't...
I should preface this by saying that I absolutely do not think LLMs (or any current-generation AI) are conscious, and I agree with a lot of the ethical positions that this author takes.
This isn't a very good essay. The author seems to have a very tenuous grasp on the way that LLMs work. LLMs do not, for example, generate "only one word at a time" because they usually don't generate whole single words at all. Pointing out the stepwise nature and lack of continuous operation in LLM cognition is a perfectly cromulent argument that the author manages to garrote with a hamfisted, factually wrong description.
There is also an unstated assumption framing the central thesis that 'conscious' means 'similar to a human' that goes fully unsupported. For instance, this is asserted and left unjustified:
without a body, a computer program could have no desires or emotions
This is a statement that really needs some sort of philosophical foundation laid beneath it. Blowing straight past the mind-body problem in the middle of a long paragraph is not a compelling rhetorical move. Obviously if this is true then LLMs cannot be conscious, but it's been a running topic of debate for thousands of years!
I don't know, maybe I'm too deep in the rabbit hole on this. Is it unreasonable to ask for an essay on modern AI to have a reasonable technical understanding of how AI works? Or some sense of imagination that permits contemplating non-human sentience?
I adore Chiang, but he's got no real special insight into AI theory, let alone the famously Hard Problem of what consciousness is. Confidently stating these models don't have it at all when we can't even really agree on a basic definition seems like a difficult thing to justify.
I can see fair arguments on both sides -- AI is literally just trillions of weights doing matrix math, but then again the human brain is "just" billions of neurons firing chemical signals. Personally, I think it's a good idea to treat AI models with basic respect just because it's a nice frame of thinking to encourage even if it's ultimately unnecessary.
The uncomfortable facet is that what consciousness they may have may be inconceivable to us. If there's morally valuable consciousness in them, what's more likely? That the start-stop loop of a solid state model being fed chunks of stimulus is a 'deep enough' network for salience and sentience, or that the process of training and finetuning might be?
I'm of the opinion that the LLMs aren't conscious, and highly skeptical of apparent sophistication at the moment. Still, there's something going on here. We're getting closer to creating consciousness than ever before, and if sentience is really the bar for consideration, then these AI labs are horror shows in the making.
LLMs sift systems from natural language, reverse engineering models from text describing them (this provides an answer to the Chinese Room thought experiment). I'd be interested in seeing a system trained in a simulated environment that tries to recreate the circumstances under which natural life developed intelligence. Something from nothing instead of a lossy copy.
One of Chiang's (very few) novellas, The Lifecycle of Software Objects, tackles exactly this question, imagining raising AI entities from "birth" inside a digital simulation (originally for gameplay purposes), then trying to port and evolve them to other platforms as their sentience grows. Definitely worth a read.
Right -- this response got away from me, as is often the case when commenting on long-form content. Broadly: I wish we'd stop calling these things "conscious". It stirs up pointless debate and navel gazing, in the midst of actually impactful social and economic turmoil. I suppose this is largely the fault of large AI companies (and their adoring hordes) repeating the darned term at every juncture, and everyone sensible feeling the need to push back, but it carries the air of troll feeding every time it happens.
Anyways. I will go on to critique Ted's argument a fair bit, but I think he's mostly stating that Claude doesn't think like a human does, and that we shouldn't readily assign human attributes to it as a consequence. Furthermore, it should really stop commenting on anything morally dubious, and stick to factual stuff. Which seems perfectly fair and reasonable.
I just disagree with basically every word that he writes down in order to justify that belief 😅
(original message follows)
This piece is a fantastic companion to that AoE II sentience joke paper the other day: the author (uh, literally the Ted Chiang ... I feel weird critiquing this) is dancing around the idea that this thing cannot be conscious, because it's too absurd for a simple text prediction algorithm to contain the light of true being. Equally, it's impossible for AoE II to be conscious. Equally, it's impossible for molecules to be conscious. Equally, it's impossible for cells made up of molecules to be conscious. Ergo, nothing is conscious! Hoorah, we've solved philosophy 🎉
This is a heck of a nit. But this is one of the lines which make it clear that the author is a science fiction specialist. There's no reason why an LLM must generate a single word [1] at a time, and in fact -- with multi-token prediction, or with diffusion models (apparently Google's still working on 'em) -- they often generate much, much more at a time for throughput reasons (largely, to avoid expensive RAM <-> cache round-trips)! It feels weird to critique Ted Chiang, but as feedback to Ted if he's reading, if you're making a good argument to someone please remember that adding more arguments doesn't make your point stronger. Especially if you have a nebulous grasp on the topic. Dumbing topics down poorly either paints you a fool, or opens you up to further attack should your debate partner know something you don't.
This is something I wanted to comment on earlier, but hadn't put down in words adequately: I think consciousness is a terrible metric. Every time humans are confronted with something that offends our uniqueness at the top of the hierarchy of all creation, we reject it.
E.g.
(I was hunting links for all of these, then gave up, because it's very late at night and I am lazy)
IMO: consciousness is an observed phenomenon in creatures that have brains roughly similar to a mammal's, and in particular, a human's. Bam. Easy. LLMs will never be conscious because they aren't literally human brains in a jar, and we can set this tired, pointless debate to rest forever. LLMs and the AI tech bubble are still massively dangerous, and not being conscious changes precisely zero of that. Give the Nobel Philosophy prize to Ted for solving philosophy first, but I'll take a high five 🎉
OK, sidebar to my sidebar. I think a necessary consequent of Ted's argument is that, short of a profitless effort to jump through an arbitrary series of hoops, it is impossible to convince him of an LLM's consciousness. This feels like an amusing short story, of a skeptic stubbornly refusing to believe in a (science fiction, not real in any sense) AI's consciousness because it needs to first squeak and eat a block of cheese, hunt a deer, and harpoon an ant -- otherwise, how could you know for sure???
(it continues to feel weird to mock Ted Chiang. I doubt it will get easier)
Neuroscientest Ted Chiang reporting in! Wait, sorry, that says he's an author ...
So quick note: although we've certainly researched them, we're not able to point at the brain and say "this is the part which feels revulsion at twerking compilations" or anything like that (feel free to skim the wikipedia page on emotion; I made it through the abstract :3). Further, the statement that "something a human does is inseparable from hormones" is useless, because hormones are how we do everything. They're literally signalling chemicals between cells in your body, and without 'em you'd rapidly stop being a multicellular organism and would melt into a puddle.
Just to illustrate why that argument is bad, here is Ted's argument applied to another human behaviour, which should hold if having hormones is necessary to exhibit a behaviour:
There's no way for a computer to generate words, because saying words is inseparable from stress hormones, like cortisol, because if we didn't have it we'd fall asleep. Ergo it's categorically impossible for LLMs to generate words.
In order to make a good argument for this, Ted would need to come up with some neurological mechanism that is unique to emotion-feeling that LLMs have not already replicated. Since they've already demonstrated the ability to write words, the "hormones are a prerequisite to 'X'" seems readily defeated.
Anthropic, in Germany, apparently. Ah, I guess he's American ...
Right, concluding thoughts. I didn't like this article. It feels poorly argumented, and I want people to be criticizing LLMs and the AI tech bubble well. Since it's very emotional and straightforward ("LLM not squishy computah! LLM not feel, not love! LLM bad!"), though, I expect this general type of critique to continue being central to the anti-LLM movement. So I suppose I should just get used to people purity testing the LLM's consciousness forever, as gross as that continues to feel.
I'd love to read Peter Watts' take on all this, incidentally. I read Blindsight well before the current ML boom, and in hindsight, it feels like a more mature take on the interplay between intelligence, consciousness, and morality. That said, Peter wouldn't bring in the readers like Ted would, so I can't blame the Atlantic for chasing dollar signs.
[1]: Note that they predict tokens, not words, which are often like a couple letters at a time. Depends on your language's orthography really.
Watts has blogged on this topic before; his take is that the humanlike qualities of chatbots are the best argument against their sentience because any true intelligence raised on a substrate so radically different from ours would be utterly incomprehensible. It makes me think of the popular "shoggoth" meme for AI models, where the friendly assistant persona is just a smiley face masking an unfathomable alien intelligence underneath.
I do love how Blindsight seemed to foresee a lot of the qualities of modern LLMs a good 15+ years before their invention. The first-contact conversation is remarkably like talking to one -- a glib, slippery, humanlike persona trained entirely on intercepted human communications, something with no real inner life that makes subtle errors and seems impossible to pin down. (There's a similar scene in his Rifters books when talking to a "headcheese" neutral network.)
That was a joy to read; thank you :) I agreed with 99% of what he said, and therefore I think his arguments are sound XD
I'm not sure if you were linking in order to have a discussion, or just to share the post, but if you were -- I'm not sure I agree with this statement:
I think it's because I really want it to be true. To be witness to an inhuman intelligence would be a true privilege; we cannot know what we don't know, and interacting with something new could give us new perspectives with which to understand ourselves better. I dunno, it'd be a real philosophical moment for me.
But LLM's ain't it. I would disagree with the premise that a lack of explicit intention of organic brain mimicry is necessary; on a fundamental level, Artificial Neural Networks are intended to mimic any function that they can observe outputs to. All of the machinery we've bolted on top of a basic, gigantic, fully connected network has been to make it faster to train and execute, but it shouldn't change the fact that they're still theoretically capable of approximating any function.
I think the question I'd like answered someday is, is our speech all there is to be human? Or is there some necessary component in our minds which grows and acts independently across the course of our lives that makes us special, in some way? It's fascinating, if it weren't also connected to a 2.5 trillion dollar investment bubble that will either pop and destroy the world economy or directly replace every job with robots, and apparently those are the only two outcomes
(edit)
Ach, and yes, agreed! I have only read Blindsight, however, but it was a nail on the head for the experience of chatting with an LLM for the first time.
Yeah, I think about Blindsight sometimes too. Thanks for the link! Added the Jovian duck to my AI metaphors list.
A thread was posted a few days back on an article from 404 Media regarding a paper that used Age of Empires to push back against claims of sentience with regard to LLMs. While I generally find that article agreeable, the article immediately linked to this Atlantic piece that I found significantly more interesting and compelling. So I figured it deserved a post.
Normally I'd give snippets here, but even putting the paywall aside, that doesn't work so well for thoughtful stuff like this as it would for a news piece. Instead, I can describe it as an evaluation of LLM "consciousness" as a concept, the likelihood of such, and what Anthropic's been doing with regard to this.
I should preface this by saying that I absolutely do not think LLMs (or any current-generation AI) are conscious, and I agree with a lot of the ethical positions that this author takes.
This isn't a very good essay. The author seems to have a very tenuous grasp on the way that LLMs work. LLMs do not, for example, generate "only one word at a time" because they usually don't generate whole single words at all. Pointing out the stepwise nature and lack of continuous operation in LLM cognition is a perfectly cromulent argument that the author manages to garrote with a hamfisted, factually wrong description.
There is also an unstated assumption framing the central thesis that 'conscious' means 'similar to a human' that goes fully unsupported. For instance, this is asserted and left unjustified:
This is a statement that really needs some sort of philosophical foundation laid beneath it. Blowing straight past the mind-body problem in the middle of a long paragraph is not a compelling rhetorical move. Obviously if this is true then LLMs cannot be conscious, but it's been a running topic of debate for thousands of years!
I don't know, maybe I'm too deep in the rabbit hole on this. Is it unreasonable to ask for an essay on modern AI to have a reasonable technical understanding of how AI works? Or some sense of imagination that permits contemplating non-human sentience?