Paper Dude made an "LLM" (just a perceptron, which is the simplest kind of neural net) using goats in the AoEII scenario creator to try to demonstrate the absurdity of calling LLMs sentient. I...
Dude made an "LLM" (just a perceptron, which is the simplest kind of neural net) using goats in the AoEII scenario creator to try to demonstrate the absurdity of calling LLMs sentient.
I think the logic doesn't fully work, probably because he's going out of his knowledge zone, but it could be much worse with this premise.
He doesn't believe it's fully absurd, but rather that the things people say are absurd in their specific formulations:
He’s also not ruling out that LLMs have some form of consciousness, but said that’s beside the point. “We tend to ascribe consciousness as some sort of binary construct (either it is or isn't!) but I'd argue that there are levels. It's hard to say that humans aren't conscious. But, what about a dog? Yes, of course. What about a potato? What about a virus? It's quite relative and we do tend to go for something human-like when we evaluate/define it, where, in reality, LLMs are things we have never seen before,” he said. “There was a recent article in science about bumblebees solving problems. Everyone was like surprisedpikachu.png — it's a great, amazing discovery. I love bumblebees and people understanding their behaviour is absolutely fantastic. But, did we really assume that they couldn't perform some sort of problem-solving? Why are we so surprised?”
The paper reads strangely to me. He painstakingly and explicitly reminds us that he’s not taking a stand on the consciousness question, but then implicitly makes it pretty clear that the AoE II...
The paper reads strangely to me. He painstakingly and explicitly reminds us that he’s not taking a stand on the consciousness question, but then implicitly makes it pretty clear that the AoE II implementation is intended as a gotcha to the people anthropomorphising LLMs in even the loosest ways.
Like… isn’t it pretty standard to assume consciousness/sentience/sapience is a function of complexity? Off the top of my head, with no rigour whatsoever, without even defining specifically what quality of mind I’m measuring (because even I don’t know), I’d still expect something like this to be a pretty uncontroversial take:
Bacteria: very unlikely
Flies: probably not
Dogs: hmmm… maybe a bit
Dolphins: yeah, getting there
Chimps: quite possibly
Humans: yes
If you managed to get >50B parameters into the AoE II implementation, I imagine the goats would start showing patterns that hit the dolphin-or-greater level on that chart and creating outputs that would get people debating the same way they do about LLMs now.
I think I get his very broad point: that the use of language pushes people to anthropomorphise more and sooner than they otherwise would. I actually agree with that point. But then he seems to see the AoE II example as a valid reductio ad absurdum when it’s actually a totally different thing (because complexity is an inherent and important part of the question), and when it probably wouldn’t be perceived as absurd at all if it were made sufficiently complex.
[Edit] Side note: not only are the replies below fascinating, they’re doing a wonderful job of underlining how happy we are to ascribe the qualities the paper talks about to biological beings without language! I think the author really did over-index on how much language is a necessity for humans to start anthropomorphising things.
Glad I'm not the only one. I feel fairly confident that flies have some sort of base level consciousness/awareness/sentience. I suppose it depends how exactly you want to define consciousness....
Glad I'm not the only one. I feel fairly confident that flies have some sort of base level consciousness/awareness/sentience. I suppose it depends how exactly you want to define consciousness. Bacteria I'm less clear on but I wouldn't feel comfortable completely ruling it out. Whereas I feel pretty comfortable saying that I don't think LLMs qualify on that scale. It seems to me that most of the research in the last 60+ years repeatedly finds that there are more and more of the things that we associate with human consciousness to be found at smaller and smaller scales in non human biological life.
I think the issue is that "sentience/sapience" is two completely different things. For sentience, yeah that's controversial. For sapience, though, I would say it's pretty uncontroversial to say...
I think the issue is that "sentience/sapience" is two completely different things. For sentience, yeah that's controversial. For sapience, though, I would say it's pretty uncontroversial to say that dogs can be smart but they aren't intelligent in the sense of "intelligent life."
Fair, this stuff is never straightforward! I guess that’s gonna depend on how we’re defining whatever quality we’re measuring, which I was deliberately fuzzy on because I don’t have the best...
Fair, this stuff is never straightforward! I guess that’s gonna depend on how we’re defining whatever quality we’re measuring, which I was deliberately fuzzy on because I don’t have the best handle on what consciousness is (do any of us, really?), let alone sentience or sapience.
It does sound like you’re on a similar page about complexity being important, though? Even if you’re putting the boundary lower than I did - maybe from a philosophical or biological difference of opinion, maybe from a semantic one on what we’re defining - the idea that there is a gradient of some kind and a reasonable cutoff point still seems to be shared.
Honestly the only thing that’s really surprised me in this thread is from @AnEarlyMartyr - the idea of putting bacteria as a stronger maybe is a slight surprise to me, not a wild one, but doing so and confidently ruling out LLMs genuinely doesn’t fit within my worldview! For the record, I’m absolutely not of the opinion that current LLMs are conscious, although I see no reason to rule out machine consciousness as a possibility in general (even if perhaps only in the far far future). I just struggle to ascribe a “maybe there’s emergent behaviour there that we don’t understand” to a biological machine like a bacterium or a fly without being forced to ascribe the same thing to a trillion parameter ball of statistical input/output processing.
I'd personally say insects and below are just biological automata. But mice/fish are where I begin to entertain some shade of gray on the consciousness scale.
I'd personally say insects and below are just biological automata. But mice/fish are where I begin to entertain some shade of gray on the consciousness scale.
I question insects on some level. I know that butterflies were proven to retain memories as caterpillars by a 10-year-old Japanese boy, who first got curious because he noticed that butterflies...
I question insects on some level. I know that butterflies were proven to retain memories as caterpillars by a 10-year-old Japanese boy, who first got curious because he noticed that butterflies he'd raised from caterpillars would fly back to him when he tried to release them. Which is super sweet and adorable, and highlights to me that the caterpillars/butterflies at least recognized him and considered him "safe".
On a personal level, I remember one stinkbug at our old house spent probably half an hour just crawling up and down a charging cable slung over the end table. It'd reach one end, turn around, crawl the other way, then turn around and repeat. It could've easily hopped off at any point after reaching a dead end, or onto the table while the cable crossed over it, but it seemed totally content to just crawl endlessly until I caught it to take outside. We had a lot of stinkbugs get in that house, but none of them behaved like that. The only explanations I can come up with are some variation of it either found that fun, or it was particularly stupid, both of which imply some level of thought/intelligence.
Of course, we may very well just be reading too much into it. Humans have an instinct to humanize everything. But who knows? Our understanding of sentience is automatically biased towards... Well, brains. Insects may have some different form of "sentience" and/or consciousness that we haven't considered.
I dunno. I have a shrimp tank and even these bottom feeders have some personality to them. Maybe some of that is just that human tendancy to humanise everything. My pet rock has some personality...
I dunno. I have a shrimp tank and even these bottom feeders have some personality to them.
Maybe some of that is just that human tendancy to humanise everything. My pet rock has some personality if you stare at it long enough.
People humanize their cats in Minecraft. I don’t think being able to see some kind of humanity in a living thing means it must have anything like a human experience.
People humanize their cats in Minecraft. I don’t think being able to see some kind of humanity in a living thing means it must have anything like a human experience.
I agree. Consciousness scales with complexity. I'm not sure exactly what that line looks like either. Even just looking at a human. I think we can agree that a single brain cell kept alive in a...
I agree. Consciousness scales with complexity. I'm not sure exactly what that line looks like either. Even just looking at a human. I think we can agree that a single brain cell kept alive in a petri dish would be unlikely to be conscious. So there must be a threshold. Humans are complex with hormone squirting glands and electric signal flinging brain cells. How much, and what parts are needed to support consciousness?
Consciousness as a function of complexity is interesting. I consider myself maximally "conscious" but I have nothing to back that up. Theoretically something sufficiently complex could become more...
Consciousness as a function of complexity is interesting. I consider myself maximally "conscious" but I have nothing to back that up. Theoretically something sufficiently complex could become more conscious than human? Is there a cap to this or does the level consciousness continue to scale with complexity to infinity?
See this is also really interesting! It’s kinda tough properly discussing something that we don’t entirely know how to define, but I was envisaging it more as a cutoff point of some kind (a...
See this is also really interesting! It’s kinda tough properly discussing something that we don’t entirely know how to define, but I was envisaging it more as a cutoff point of some kind (a necessarily fuzzy one, as all things are when they hit reality, but a cutoff nonetheless) where we err on the side of caution in the grey area but largely define above the line as above the line in a binary sense.
But then I suppose I am inherently considering a gradient on some axis too (possibly a different axis?) because I have different concerns about what I’d consider ethical treatment of creatures at different points on the list. It’s a fascinating topic, that’s for sure!
Even humans are not really maximally conscious. A lot of our thought processes are done unconsciously, like language generation and most decision making. We are only aware of a portion of our...
Even humans are not really maximally conscious. A lot of our thought processes are done unconsciously, like language generation and most decision making. We are only aware of a portion of our internal processing, and even that awareness is pretty inconsistent (e.g., blanking out while driving your regular commute).
I don't think we actually know that complexity is the right metric. I mean, I think it's undeniable that with greater complexity you can make something that seems more conscious; but just because...
I don't think we actually know that complexity is the right metric. I mean, I think it's undeniable that with greater complexity you can make something that seems more conscious; but just because something seems more conscious, or can convince more people, doesn't necessarily mean that it is conscious.
There was a paper a couple months back that presented possible evidence of quantum effects in the brain. If that's true (and I have no idea) then it could be that consciousness doesn't actually require all that much complexity, but it does require these weird quantum effects.
That being said, maybe it doesn't really matter? If something is complex enough that it can do useful stuff, then whether it is conscious is really more of a philosophical / ethical question.
Was that Roger Penrose, by any chance? I was fortunate enough to be at one of his talks a few years back, and he dipped into theory of consciousness among a lot of other fascinating topics. I can...
Was that Roger Penrose, by any chance? I was fortunate enough to be at one of his talks a few years back, and he dipped into theory of consciousness among a lot of other fascinating topics. I can certainly believe the conceptual idea that as-yet-unexplored quantum effects, and/or higher-order interactions in the structure of the electrical fields of the neurons far beyond the signal:response “circuitry” we can currently analyse could be crucial.
We’re definitely in deep “nobody really knows yet” territory, which I always find exciting! I’ll admit I find it hard to conceptualise a “brain” that could trigger these kind of interactions we don’t even yet understand without being fairly complex - even if complex in ways we can’t yet measure - but as you say, we don’t know anything for sure right now.
There is a competing view that maybe everything actually is conscious, but if you dont think about it then it doesnt really matter in a meaningful way. Like, maybe rocks are actually conscious but...
There is a competing view that maybe everything actually is conscious, but if you dont think about it then it doesnt really matter in a meaningful way.
Like, maybe rocks are actually conscious but since they dont have the capacity to think they cant really suffer or fall in love or anything so nobody cares, its just an eternity of being aware of heat or pressure or whatnot.
And then as you go into various forms of life you develop sophisticated systems for sensing and responding to the environment, and that consciousness starts mattering more because it somehow couples into those complex systems and affects how living things behave.
And then eventually at the tier of life humans are at, you develop self awareness and that amplifies the significance of consciousness in some sense, which is why we think of there being a cutoff between chimps and ants.
I’m not sure how implementing a perception in AoE is supposed to indicate anything. Do they not know what a Turing machine is? You can often contrive ways for programs running on a Turing machine...
I’m not sure how implementing a perception in AoE is supposed to indicate anything.
Do they not know what a Turing machine is? You can often contrive ways for programs running on a Turing machine to themselves be Turing complete. Many video games apply. So do unexpected things like CSS or the c++ preprocessor.
Unless you believe it is completely impossible for a Turing machine to represent consciousness, then it will be possible theoretically to construct said machine consciousness in Minecraft or whatever, because Minecraft is Turing complete. It do be like that.
I've been increasingly annoyed with coverage from 404 Media on the topic of AI. It just seems to me that they're pessimistic and gleefully contrarian on the subject despite the clear advances in...
I've been increasingly annoyed with coverage from 404 Media on the topic of AI. It just seems to me that they're pessimistic and gleefully contrarian on the subject despite the clear advances in the field. I've not seen them lose one kind word about modern attention-aware transformers.
Of course, they're not obliged to, anyone can write what they want, but it does to me paint a pattern. They're just all too ready to jump on any story that even slightly antagonises "AI", as if that means anything.
This article in specific requires a broken premise. I don't think any researcher worth their weight in salt is saying that any LLM currently in development or on the market is, has ever been, or can be conscious.
I don't think I need to expand on this, but if anyone's curious, here's a collection of articles that I thought were particularly uncharitable and really only read to me as gawking at the (admittedly numerous and very funny) failures symptomatic of an industry in a state of panicked flailing.
I had discovered 404 via artists on Fedi, so if they exist at that intersection of art and technology the hostilities toward genAI in particular make sense. But I also don't understand how you...
I had discovered 404 via artists on Fedi, so if they exist at that intersection of art and technology the hostilities toward genAI in particular make sense.
But I also don't understand how you consider that Trump one in particular uncharitable at all. Best I can tell it's not even criticizing the tech so much as the ignoramus in chief and companies kowtowing to his bigotry.
There is also something to be said that they do substantial investigative journalism. And generally speaking, that means reporting on the dirty underbelly than the squeaky clean PR releases.
Paper
Dude made an "LLM" (just a perceptron, which is the simplest kind of neural net) using goats in the AoEII scenario creator to try to demonstrate the absurdity of calling LLMs sentient.
I think the logic doesn't fully work, probably because he's going out of his knowledge zone, but it could be much worse with this premise.
He doesn't believe it's fully absurd, but rather that the things people say are absurd in their specific formulations:
The paper reads strangely to me. He painstakingly and explicitly reminds us that he’s not taking a stand on the consciousness question, but then implicitly makes it pretty clear that the AoE II implementation is intended as a gotcha to the people anthropomorphising LLMs in even the loosest ways.
Like… isn’t it pretty standard to assume consciousness/sentience/sapience is a function of complexity? Off the top of my head, with no rigour whatsoever, without even defining specifically what quality of mind I’m measuring (because even I don’t know), I’d still expect something like this to be a pretty uncontroversial take:
If you managed to get >50B parameters into the AoE II implementation, I imagine the goats would start showing patterns that hit the dolphin-or-greater level on that chart and creating outputs that would get people debating the same way they do about LLMs now.
I think I get his very broad point: that the use of language pushes people to anthropomorphise more and sooner than they otherwise would. I actually agree with that point. But then he seems to see the AoE II example as a valid reductio ad absurdum when it’s actually a totally different thing (because complexity is an inherent and important part of the question), and when it probably wouldn’t be perceived as absurd at all if it were made sufficiently complex.
[Edit] Side note: not only are the replies below fascinating, they’re doing a wonderful job of underlining how happy we are to ascribe the qualities the paper talks about to biological beings without language! I think the author really did over-index on how much language is a necessity for humans to start anthropomorphising things.
I think that's pretty controversial! For flies I would say "probably", and for everything above I would say "certainly".
Glad I'm not the only one. I feel fairly confident that flies have some sort of base level consciousness/awareness/sentience. I suppose it depends how exactly you want to define consciousness. Bacteria I'm less clear on but I wouldn't feel comfortable completely ruling it out. Whereas I feel pretty comfortable saying that I don't think LLMs qualify on that scale. It seems to me that most of the research in the last 60+ years repeatedly finds that there are more and more of the things that we associate with human consciousness to be found at smaller and smaller scales in non human biological life.
I think the issue is that "sentience/sapience" is two completely different things. For sentience, yeah that's controversial. For sapience, though, I would say it's pretty uncontroversial to say that dogs can be smart but they aren't intelligent in the sense of "intelligent life."
Fair, this stuff is never straightforward! I guess that’s gonna depend on how we’re defining whatever quality we’re measuring, which I was deliberately fuzzy on because I don’t have the best handle on what consciousness is (do any of us, really?), let alone sentience or sapience.
It does sound like you’re on a similar page about complexity being important, though? Even if you’re putting the boundary lower than I did - maybe from a philosophical or biological difference of opinion, maybe from a semantic one on what we’re defining - the idea that there is a gradient of some kind and a reasonable cutoff point still seems to be shared.
Honestly the only thing that’s really surprised me in this thread is from @AnEarlyMartyr - the idea of putting bacteria as a stronger maybe is a slight surprise to me, not a wild one, but doing so and confidently ruling out LLMs genuinely doesn’t fit within my worldview! For the record, I’m absolutely not of the opinion that current LLMs are conscious, although I see no reason to rule out machine consciousness as a possibility in general (even if perhaps only in the far far future). I just struggle to ascribe a “maybe there’s emergent behaviour there that we don’t understand” to a biological machine like a bacterium or a fly without being forced to ascribe the same thing to a trillion parameter ball of statistical input/output processing.
I'd personally say insects and below are just biological automata. But mice/fish are where I begin to entertain some shade of gray on the consciousness scale.
I question insects on some level. I know that butterflies were proven to retain memories as caterpillars by a 10-year-old Japanese boy, who first got curious because he noticed that butterflies he'd raised from caterpillars would fly back to him when he tried to release them. Which is super sweet and adorable, and highlights to me that the caterpillars/butterflies at least recognized him and considered him "safe".
On a personal level, I remember one stinkbug at our old house spent probably half an hour just crawling up and down a charging cable slung over the end table. It'd reach one end, turn around, crawl the other way, then turn around and repeat. It could've easily hopped off at any point after reaching a dead end, or onto the table while the cable crossed over it, but it seemed totally content to just crawl endlessly until I caught it to take outside. We had a lot of stinkbugs get in that house, but none of them behaved like that. The only explanations I can come up with are some variation of it either found that fun, or it was particularly stupid, both of which imply some level of thought/intelligence.
Of course, we may very well just be reading too much into it. Humans have an instinct to humanize everything. But who knows? Our understanding of sentience is automatically biased towards... Well, brains. Insects may have some different form of "sentience" and/or consciousness that we haven't considered.
I dunno. I have a shrimp tank and even these bottom feeders have some personality to them.
Maybe some of that is just that human tendancy to humanise everything. My pet rock has some personality if you stare at it long enough.
People humanize their cats in Minecraft. I don’t think being able to see some kind of humanity in a living thing means it must have anything like a human experience.
I agree. Consciousness scales with complexity. I'm not sure exactly what that line looks like either. Even just looking at a human. I think we can agree that a single brain cell kept alive in a petri dish would be unlikely to be conscious. So there must be a threshold. Humans are complex with hormone squirting glands and electric signal flinging brain cells. How much, and what parts are needed to support consciousness?
Consciousness as a function of complexity is interesting. I consider myself maximally "conscious" but I have nothing to back that up. Theoretically something sufficiently complex could become more conscious than human? Is there a cap to this or does the level consciousness continue to scale with complexity to infinity?
See this is also really interesting! It’s kinda tough properly discussing something that we don’t entirely know how to define, but I was envisaging it more as a cutoff point of some kind (a necessarily fuzzy one, as all things are when they hit reality, but a cutoff nonetheless) where we err on the side of caution in the grey area but largely define above the line as above the line in a binary sense.
But then I suppose I am inherently considering a gradient on some axis too (possibly a different axis?) because I have different concerns about what I’d consider ethical treatment of creatures at different points on the list. It’s a fascinating topic, that’s for sure!
Even humans are not really maximally conscious. A lot of our thought processes are done unconsciously, like language generation and most decision making. We are only aware of a portion of our internal processing, and even that awareness is pretty inconsistent (e.g., blanking out while driving your regular commute).
I don't think we actually know that complexity is the right metric. I mean, I think it's undeniable that with greater complexity you can make something that seems more conscious; but just because something seems more conscious, or can convince more people, doesn't necessarily mean that it is conscious.
There was a paper a couple months back that presented possible evidence of quantum effects in the brain. If that's true (and I have no idea) then it could be that consciousness doesn't actually require all that much complexity, but it does require these weird quantum effects.
That being said, maybe it doesn't really matter? If something is complex enough that it can do useful stuff, then whether it is conscious is really more of a philosophical / ethical question.
Was that Roger Penrose, by any chance? I was fortunate enough to be at one of his talks a few years back, and he dipped into theory of consciousness among a lot of other fascinating topics. I can certainly believe the conceptual idea that as-yet-unexplored quantum effects, and/or higher-order interactions in the structure of the electrical fields of the neurons far beyond the signal:response “circuitry” we can currently analyse could be crucial.
We’re definitely in deep “nobody really knows yet” territory, which I always find exciting! I’ll admit I find it hard to conceptualise a “brain” that could trigger these kind of interactions we don’t even yet understand without being fairly complex - even if complex in ways we can’t yet measure - but as you say, we don’t know anything for sure right now.
There is a competing view that maybe everything actually is conscious, but if you dont think about it then it doesnt really matter in a meaningful way.
Like, maybe rocks are actually conscious but since they dont have the capacity to think they cant really suffer or fall in love or anything so nobody cares, its just an eternity of being aware of heat or pressure or whatnot.
And then as you go into various forms of life you develop sophisticated systems for sensing and responding to the environment, and that consciousness starts mattering more because it somehow couples into those complex systems and affects how living things behave.
And then eventually at the tier of life humans are at, you develop self awareness and that amplifies the significance of consciousness in some sense, which is why we think of there being a cutoff between chimps and ants.
I’m not sure how implementing a perception in AoE is supposed to indicate anything.
Do they not know what a Turing machine is? You can often contrive ways for programs running on a Turing machine to themselves be Turing complete. Many video games apply. So do unexpected things like CSS or the c++ preprocessor.
Unless you believe it is completely impossible for a Turing machine to represent consciousness, then it will be possible theoretically to construct said machine consciousness in Minecraft or whatever, because Minecraft is Turing complete. It do be like that.
I've been increasingly annoyed with coverage from 404 Media on the topic of AI. It just seems to me that they're pessimistic and gleefully contrarian on the subject despite the clear advances in the field. I've not seen them lose one kind word about modern attention-aware transformers.
Of course, they're not obliged to, anyone can write what they want, but it does to me paint a pattern. They're just all too ready to jump on any story that even slightly antagonises "AI", as if that means anything.
This article in specific requires a broken premise. I don't think any researcher worth their weight in salt is saying that any LLM currently in development or on the market is, has ever been, or can be conscious.
I don't think I need to expand on this, but if anyone's curious, here's a collection of articles that I thought were particularly uncharitable and really only read to me as gawking at the (admittedly numerous and very funny) failures symptomatic of an industry in a state of panicked flailing.
It Is Trivially Easy to Use Reddit to Manipulate AI Search, Research Suggests
Software Developers Say AI Is Rotting Their Brains
Anthropic Promises Trump Admin Its AI Is Not Woke
I had discovered 404 via artists on Fedi, so if they exist at that intersection of art and technology the hostilities toward genAI in particular make sense.
But I also don't understand how you consider that Trump one in particular uncharitable at all. Best I can tell it's not even criticizing the tech so much as the ignoramus in chief and companies kowtowing to his bigotry.
There is also something to be said that they do substantial investigative journalism. And generally speaking, that means reporting on the dirty underbelly than the squeaky clean PR releases.
Hopefully this doesnt end up being a schrodingers box scenario.