Moreover and more importantly, we won’t know before we’ve already manufactured thousands or millions of disputably conscious AI systems. Engineering sprints ahead while consciousness science lags. Consciousness scientists – and philosophers, and policy-makers, and the public – are watching AI development disappear over the hill. Soon we will hear a voice shout back to us, “Now I am just as conscious, just as full of experience and feeling, as any human”, and we won’t know whether to believe it. We will need to decide, as individuals and as a society, whether to treat AI systems as conscious, nonconscious, semi-conscious, or incomprehensibly alien, before we have adequate grounds to justify that decision.
...
In this book, I aim to convince you that the experts do not know, and you do not know,
and society collectively does not and will not know, and all is fog.
How would that work? Feeling is a complicated biochemical process rather than a side effect of intelligence. Intelligence and feeling do not somehow spontaneously come in the same package. In...
“Now I am just as conscious, just as full of experience and feeling, as any human”
How would that work? Feeling is a complicated biochemical process rather than a side effect of intelligence. Intelligence and feeling do not somehow spontaneously come in the same package.
In order to replicate feeling it would need to be the explicit goal, it won't happen accidentally. The idea that we "won't know" is fun for science fiction, but with current technology the accurate sentence is "won't happen".
Maybe someday, but only as a result of an intentional initiative. Even if we imagine recursive self improvement, feeling wouldn't be a logical target, we'd need to ask for it
Why not? We don't even know what "feeling" physically is. The reason we don't attribute feelings to a toaster or a rock is because it would make life too complicated, not because we can point a...
In order to replicate feeling it would need to be the explicit goal, it won't happen accidentally.
Why not? We don't even know what "feeling" physically is. The reason we don't attribute feelings to a toaster or a rock is because it would make life too complicated, not because we can point a feel-o-meter at them and it reads 0.0.
Even if you believe that some kind of god or other metaphysical entity breathes qualia into dead objects: As long as we don't know how that works, we cannot know that we haven't accidentally replicated that process.
We don't know what feeling physically is in a comprehensive sense. But we have quantified enough parts of feeling to confidently say that it's complicated, it's both neurological and biochemical,...
We don't know what feeling physically is in a comprehensive sense. But we have quantified enough parts of feeling to confidently say that it's complicated, it's both neurological and biochemical, it's not exclusively controlled or originated by the brain, and it evolved through natural selection to help organisms behave in ways that would cause them to have a better chance of survival.
We don't attribute feelings to toasters and rocks because they lack both the equipment and the impetus to produce feelings, not because it would be inconvenient. It's a bit muddier when it comes to, for example, insects.
Do we know that? I would think we don't know enough about "feelings" or intelligence to say anything like that. Ive heard it argued intelligence is a side effect of feelings. what do any of these...
Feeling is a complicated biochemical process rather than a side effect of intelligence. Intelligence and feeling do not somehow spontaneously come in the same package.
Do we know that? I would think we don't know enough about "feelings" or intelligence to say anything like that. Ive heard it argued intelligence is a side effect of feelings. what do any of these things mean?
I "feel" like LLMs are no where near consciousness at this point.
I feel like they might be as intelligent as just the language center of a brain, cut out and dropped in a vat.
That seems like a good place for me to stop and realize that I cant trust my judgement on this at all, because ultimately, I don't understand how that language center works, or what consciousness or intelligence or feelings are.
"The experts do not know, and you do not know, and society collectively does not and will not know, and all is fog"
I think that's the exact right thing that everyone needs to know right now. Nobody knows what consciousness or intelligence is. Any tech companies confidence in anything at this point is all bullshit. The insistence that any large neural network running in vram is definitely NOT conscious, or x, y, or z, is also bullshit because no one knows how they work. Same as some one insisting that slime mold isn't conscious. Its smarter than we are at "some" stuff.
To be clear there are experts we should listen to and consider their knowledge. People who understand neurology or linear algebra way better than I do. but the engineers that are working on this will tell you the actual mechanism within these matrices is a giant black box. We understand how slime mold or humans send signals around their thinking systems, and maybe a little bit about their general structure. We cant follow the actual program that runs on any of this stuff.
We are running neural networks that program themselves with code that is more complicated than we can currently understand. We should treat all this stuff like an alien organism we found on an asteroid. Or a GMO slime mold that we can somehow train to talk to us. We cant know what it is yet. We know it can do some novel stuff we haven't seen before, they are probably mostly parlor tricks, but how can we be sure?
How is it possible that some pink slime in my skull that is just self replicating chemical reactions thinks the way humans do? So confident of self awareness? As far as we can tell it just self organized that way over billions of years of trial and error and dense memory storage.
Can we recreate something similar by forcing petabytes of data into machines that cycle 2 billions times a second for years? Probably not. Especially when the goal is a product like the current assistants. But it feels like the base models themselves might have a depth of knowledge that is just barely scratched by this current post-train assistant, run once architecture.
We connect up our "language centers" to other neural networks, let them save active memories and spawn new agents to act on them on regular cycles. we'll keep adding layers and connections and experimenting. It will still probably be nothing like humans. But what will it become? (What rough beast?) I think nobody knows. Its exciting and terrifying at the same time. And its at least plausible that all this will play out in our lifetimes. I think its worth examining now.
Maybe it would happen because it's learning from people or animals somehow? LLM's learn a lot of different ways of writing from what people have written.
Maybe it would happen because it's learning from people or animals somehow? LLM's learn a lot of different ways of writing from what people have written.
I think your question amounts to asking whether AI models can "feel" or merely mimic feeling , which is addressed in Chapter 7 ("The Mimicry Argument Against AI Consciousness"). First, it's worth...
I think your question amounts to asking whether AI models can "feel" or merely mimic feeling , which is addressed in Chapter 7 ("The Mimicry Argument Against AI Consciousness"). First, it's worth establishing what the author means by mimicry; these excerpts are probably sufficient for you to get the drift.
When you know that something has been designed or has evolved as a mimic, you cannot infer from the readily observed feature to the further feature in the way you ordinarily would in the model. At least you can’t do so without further evidence. Once you know that the viceroy [butterfly] mimics the monarch, you cannot infer from its wing pattern to its toxicity. Maybe the viceroy is toxic, but that would need to be separately established. Similarly, knowing that [a toy doll's] “hello” mimics a human greeting, you cannot infer that the toy actually intends to greet you.
[...]
A consciousness mimic is an entity that mimics some superficial or readily observable features that, in some set of model entities, reliably indicate consciousness. But because the mimic has been designed or selected specifically to display those superficial features, we the receivers cannot justifiably infer underlying consciousness – not in the same way we can when we see those same features in the model entity. This is obvious for the “hello” toy, less obvious but still true for entities specifically designed to pass the Turing test or otherwise mimic the surface features of human language. An important class of AI systems are consciousness mimics in this sense.
Searle and Bender aim for a stronger conclusion, inviting us positively to conclude that the mimics do not have conscious linguistic understanding. I don’t think we can know this from their arguments. But both thought experiments successfully describe consciousness mimics whose outputs we should reasonably mistrust. The case for consciousness is undercut. It does not follow that the case against consciousness is established.
[...]
Ordinarily, if you’re having what seems to be a meaningful conversation, you can infer that your conversation partner is conscious and understands the meaning of your words. But if you know that the entity is designed to mimic human text outputs, you ought no longer be so sure.
The author mentions later in the chapter that "the large majority of experts on consciousness agree that classic pure transformers are not conscious", so if a current generation LLM were to declare, "Now I am just as conscious, just as full of experience and feeling, as any human", we are probably correct to dismiss the statement as mimicry. However, post-training complicates this assessment: tomorrow's frontier models will not be evaluated exclusively on their ability to produce plausible-sounding text. Indeed, "feeling" will likely be an explicit (or at least auxiliary) training goal (emphasis added):
Recent Large Language Models such as ChatGPT and Claude build upon the mimicry structures of pure transformer models but also receive post-training. Reinforcement learning from human feedback “rewards” human-approved outputs, strengthening the associated weights. Models can also be reinforced for being “right” by external standards, and some can access tools like calculators. To the extent the machines move beyond pure mimicry, the Mimicry Argument applies less straightforwardly. For now, mimicry-based skepticism still seems warranted, since their core architecture remains close to that of pure transformers, and their humanlike outputs are still best explained by their pretraining on word co-occurrence in human texts.
In the longer term, we might imagine architectures more thoroughly trained on the rights and wrongs of the world itself – maybe like AlphaGo but with the larger world, or some significant portion of it, as its playground. Outputs would be shaped primarily by success in real-world complex tasks, perhaps including communicative tasks, rather than by resemblance to humans. The Mimicry Argument would then no longer apply. Skepticism about their consciousness, if warranted, would need a different basis.
In general, this book is quite comprehensive and comprehensible. There is also an entire chapter on whether biological substrate matters (chapter 10), which will likely address your question from another angle.
Calling this "skeptical" seems disingenuous. I am skeptical that an LLM is anywhere tangental in the process of becoming what we colloquially believe AI to be, never-mind it becoming conscious. If...
Calling this "skeptical" seems disingenuous. I am skeptical that an LLM is anywhere tangental in the process of becoming what we colloquially believe AI to be, never-mind it becoming conscious. If anything an LLM could be hooked up to some form of AI in order for it to speak our language, but nothing more.
So if a current generation LLM were to declare, "Now I am just as conscious, just as full of experience and feeling, as any human", we are probably correct to dismiss the statement as mimicry.
Suppose we take the argument at face value. There is no LLM declaring this, so the argument is entirely moot. It can repeat those words when prompted to do so, to role-play as a character who says that, for instance. But this output doesn't change the LLM, it doesn't feed back into it and propagate throughout. Even "thinking" models that can feed outputs into inputs and reprocess them to give additional output isn't ultimately building its understanding of the world. LLMs lack the basic premise of even being a constant 'thing'. When I type something into Claude, it has no bearing on the millions of other prompts it is answering. We are not communicating with the same consciousness running on servers across the country. That's just not how large language models work.
If anyone starts having unprompted conversations with an AI LLM that are actually in any way stating that it is conscious, and it persist over time, then maybe it would be worthwhile to start having the conversations that people want to have over AI consciousness. But as it stands, I really see no path from LLM to actual persistent intelligence. It's not what LLMs are even designed to do. It feels like asking when protein folding simulations are going to generate new lifeforms.
I'd love to be wrong, honestly. But it's just not what we are designing in any way.
My understanding of the author's use of "skeptical" is not that we should be skeptical that AI is conscious, but rather we should be skeptical of anyone who claims to know whether AI is/can become...
My understanding of the author's use of "skeptical" is not that we should be skeptical that AI is conscious, but rather we should be skeptical of anyone who claims to know whether AI is/can become conscious or not. That is, we should be skeptical of arguments both for and against AI consciousness. If you haven't yet, you should really read the first couple chapters in which the author elaborates on how "consciousness" and "artificial intelligence" are load-bearing terms that are basically impossible to nail down. (As if to prove the point, some philosophers argue that rocks are conscious.) When it comes to consciousness, there are no obvious answers, just as there is no obvious divide between "artificial intelligence" and "real intelligence", so to speak.
Suppose we take the argument at face value. There is no LLM declaring this, so the argument is entirely moot.
Well, as I said, the author is not so much concerned about current generation LLMs as he is future iterations. Nevertheless, it's worth examining some of the points you made. You are declaring that consciousness must contain some "essential" properties (see chapters 3 and 4), specifically that it must have "access" and "specious presence" (see text or my footnote [1]). However, neither of these properties are necessarily essential. For instance, it could be that we have conscious experiences that are not available for further processing (experiences that don't backpropagate, so to speak), but because we wouldn't process these experiences, we wouldn't remember them, either
Moreover, generalizing any essential property risks suffering from a certain type of "sampling bias", as the author calls it -- we risk making a classification error in which we equivocate between "consciousness" (whatever that is) to the human experience of having consciousness. For example, it could be that some creatures experience the world in a way outside time (like Kurt Vonnegut's Tralfamadorians or Ted Chiang's Heptapods), which would render "specious presence" unnecessary.
In general, I would really just recommend you read Schwitzgebel's book. He's certainly thought more about this than either or us.
[1] Taken from chapter 3:
(4) Access. To be conscious, an experience must be available for “downstream” cognitive processes like inference and planning, verbal report, and memory. No conscious experience can simply occur in a cognitive dead end, with no possible further cognitive consequences.
(9) Specious presence. All conscious experiences are felt to be temporally extended, smeared across a small interval of time (a fraction of a second to a few seconds) – generally called the “specious present” – rather than being strictly instantaneous or wholly atemporal.
This captures some of my sentiment about consciousness conversations around LLMs. You first have to misunderstand the technology (willfully or not) and then speculate heavily about what could...
This captures some of my sentiment about consciousness conversations around LLMs. You first have to misunderstand the technology (willfully or not) and then speculate heavily about what could happen in the future once being forced to acknowledge that current tech almost definitely isn't headed there. It's science fiction, which I generally think is great, but it's often framed as rational or academic when it's neither.
If a paper, or in this case a book, were to lead with "Here's how the technology works... Given how it works, even with our incomplete understanding of what consciousness is, it's safe to say that consciousness is not going to happen here. With that in mind, let's speculate about what technology that reasonably could lead in that direction might look like... That I could appreciate.
But so far what keeps coming out is attempts to overfit LLM tech into a path to consciousness because a significant percentage of the population really wants conscious machines.
I think in general its good to err on the side of open mindedness, when addressing our knowledge of consciousness. The books theme seems to be " The experts do not know, and you do not know, and...
I think in general its good to err on the side of open mindedness, when addressing our knowledge of consciousness.
The books theme seems to be " The experts do not know, and you do not know,
and society collectively does not and will not know, and all is fog"
That feels like my definition of the word skepticism.
It's hard to read when the framing is based on the idea that there's a reasonable debate to be had about AI and consciousness. To even begin to have that conversation you need to first make a case...
It's hard to read when the framing is based on the idea that there's a reasonable debate to be had about AI and consciousness. To even begin to have that conversation you need to first make a case for the possibility that any technology we currently have could be advanced to the point that it could result in consciousness. Which presumably would first require AGI. That's a big ask.
And without that the whole premise falls apart and it's just speculation for its own sake.
From the book manuscript:
...
How would that work? Feeling is a complicated biochemical process rather than a side effect of intelligence. Intelligence and feeling do not somehow spontaneously come in the same package.
In order to replicate feeling it would need to be the explicit goal, it won't happen accidentally. The idea that we "won't know" is fun for science fiction, but with current technology the accurate sentence is "won't happen".
Maybe someday, but only as a result of an intentional initiative. Even if we imagine recursive self improvement, feeling wouldn't be a logical target, we'd need to ask for it
Why not? We don't even know what "feeling" physically is. The reason we don't attribute feelings to a toaster or a rock is because it would make life too complicated, not because we can point a feel-o-meter at them and it reads 0.0.
Even if you believe that some kind of god or other metaphysical entity breathes qualia into dead objects: As long as we don't know how that works, we cannot know that we haven't accidentally replicated that process.
We don't know what feeling physically is in a comprehensive sense. But we have quantified enough parts of feeling to confidently say that it's complicated, it's both neurological and biochemical, it's not exclusively controlled or originated by the brain, and it evolved through natural selection to help organisms behave in ways that would cause them to have a better chance of survival.
We don't attribute feelings to toasters and rocks because they lack both the equipment and the impetus to produce feelings, not because it would be inconvenient. It's a bit muddier when it comes to, for example, insects.
Do we know that? I would think we don't know enough about "feelings" or intelligence to say anything like that. Ive heard it argued intelligence is a side effect of feelings. what do any of these things mean?
I "feel" like LLMs are no where near consciousness at this point.
I feel like they might be as intelligent as just the language center of a brain, cut out and dropped in a vat.
That seems like a good place for me to stop and realize that I cant trust my judgement on this at all, because ultimately, I don't understand how that language center works, or what consciousness or intelligence or feelings are.
"The experts do not know, and you do not know, and society collectively does not and will not know, and all is fog"
I think that's the exact right thing that everyone needs to know right now. Nobody knows what consciousness or intelligence is. Any tech companies confidence in anything at this point is all bullshit. The insistence that any large neural network running in vram is definitely NOT conscious, or x, y, or z, is also bullshit because no one knows how they work. Same as some one insisting that slime mold isn't conscious. Its smarter than we are at "some" stuff.
To be clear there are experts we should listen to and consider their knowledge. People who understand neurology or linear algebra way better than I do. but the engineers that are working on this will tell you the actual mechanism within these matrices is a giant black box. We understand how slime mold or humans send signals around their thinking systems, and maybe a little bit about their general structure. We cant follow the actual program that runs on any of this stuff.
We are running neural networks that program themselves with code that is more complicated than we can currently understand. We should treat all this stuff like an alien organism we found on an asteroid. Or a GMO slime mold that we can somehow train to talk to us. We cant know what it is yet. We know it can do some novel stuff we haven't seen before, they are probably mostly parlor tricks, but how can we be sure?
How is it possible that some pink slime in my skull that is just self replicating chemical reactions thinks the way humans do? So confident of self awareness? As far as we can tell it just self organized that way over billions of years of trial and error and dense memory storage.
Can we recreate something similar by forcing petabytes of data into machines that cycle 2 billions times a second for years? Probably not. Especially when the goal is a product like the current assistants. But it feels like the base models themselves might have a depth of knowledge that is just barely scratched by this current post-train assistant, run once architecture.
We connect up our "language centers" to other neural networks, let them save active memories and spawn new agents to act on them on regular cycles. we'll keep adding layers and connections and experimenting. It will still probably be nothing like humans. But what will it become? (What rough beast?) I think nobody knows. Its exciting and terrifying at the same time. And its at least plausible that all this will play out in our lifetimes. I think its worth examining now.
Maybe it would happen because it's learning from people or animals somehow? LLM's learn a lot of different ways of writing from what people have written.
I think your question amounts to asking whether AI models can "feel" or merely mimic feeling , which is addressed in Chapter 7 ("The Mimicry Argument Against AI Consciousness"). First, it's worth establishing what the author means by mimicry; these excerpts are probably sufficient for you to get the drift.
The author mentions later in the chapter that "the large majority of experts on consciousness agree that classic pure transformers are not conscious", so if a current generation LLM were to declare, "Now I am just as conscious, just as full of experience and feeling, as any human", we are probably correct to dismiss the statement as mimicry. However, post-training complicates this assessment: tomorrow's frontier models will not be evaluated exclusively on their ability to produce plausible-sounding text. Indeed, "feeling" will likely be an explicit (or at least auxiliary) training goal (emphasis added):
In general, this book is quite comprehensive and comprehensible. There is also an entire chapter on whether biological substrate matters (chapter 10), which will likely address your question from another angle.
Calling this "skeptical" seems disingenuous. I am skeptical that an LLM is anywhere tangental in the process of becoming what we colloquially believe AI to be, never-mind it becoming conscious. If anything an LLM could be hooked up to some form of AI in order for it to speak our language, but nothing more.
Suppose we take the argument at face value. There is no LLM declaring this, so the argument is entirely moot. It can repeat those words when prompted to do so, to role-play as a character who says that, for instance. But this output doesn't change the LLM, it doesn't feed back into it and propagate throughout. Even "thinking" models that can feed outputs into inputs and reprocess them to give additional output isn't ultimately building its understanding of the world. LLMs lack the basic premise of even being a constant 'thing'. When I type something into Claude, it has no bearing on the millions of other prompts it is answering. We are not communicating with the same consciousness running on servers across the country. That's just not how large language models work.
If anyone starts having unprompted conversations with an AI LLM that are actually in any way stating that it is conscious, and it persist over time, then maybe it would be worthwhile to start having the conversations that people want to have over AI consciousness. But as it stands, I really see no path from LLM to actual persistent intelligence. It's not what LLMs are even designed to do. It feels like asking when protein folding simulations are going to generate new lifeforms.
I'd love to be wrong, honestly. But it's just not what we are designing in any way.
My understanding of the author's use of "skeptical" is not that we should be skeptical that AI is conscious, but rather we should be skeptical of anyone who claims to know whether AI is/can become conscious or not. That is, we should be skeptical of arguments both for and against AI consciousness. If you haven't yet, you should really read the first couple chapters in which the author elaborates on how "consciousness" and "artificial intelligence" are load-bearing terms that are basically impossible to nail down. (As if to prove the point, some philosophers argue that rocks are conscious.) When it comes to consciousness, there are no obvious answers, just as there is no obvious divide between "artificial intelligence" and "real intelligence", so to speak.
Well, as I said, the author is not so much concerned about current generation LLMs as he is future iterations. Nevertheless, it's worth examining some of the points you made. You are declaring that consciousness must contain some "essential" properties (see chapters 3 and 4), specifically that it must have "access" and "specious presence" (see text or my footnote [1]). However, neither of these properties are necessarily essential. For instance, it could be that we have conscious experiences that are not available for further processing (experiences that don't backpropagate, so to speak), but because we wouldn't process these experiences, we wouldn't remember them, either
Moreover, generalizing any essential property risks suffering from a certain type of "sampling bias", as the author calls it -- we risk making a classification error in which we equivocate between "consciousness" (whatever that is) to the human experience of having consciousness. For example, it could be that some creatures experience the world in a way outside time (like Kurt Vonnegut's Tralfamadorians or Ted Chiang's Heptapods), which would render "specious presence" unnecessary.
In general, I would really just recommend you read Schwitzgebel's book. He's certainly thought more about this than either or us.
[1] Taken from chapter 3:
This captures some of my sentiment about consciousness conversations around LLMs. You first have to misunderstand the technology (willfully or not) and then speculate heavily about what could happen in the future once being forced to acknowledge that current tech almost definitely isn't headed there. It's science fiction, which I generally think is great, but it's often framed as rational or academic when it's neither.
If a paper, or in this case a book, were to lead with "Here's how the technology works... Given how it works, even with our incomplete understanding of what consciousness is, it's safe to say that consciousness is not going to happen here. With that in mind, let's speculate about what technology that reasonably could lead in that direction might look like... That I could appreciate.
But so far what keeps coming out is attempts to overfit LLM tech into a path to consciousness because a significant percentage of the population really wants conscious machines.
I think in general its good to err on the side of open mindedness, when addressing our knowledge of consciousness.
The books theme seems to be " The experts do not know, and you do not know,
and society collectively does not and will not know, and all is fog"
That feels like my definition of the word skepticism.
More detail on my above post
It's hard to read when the framing is based on the idea that there's a reasonable debate to be had about AI and consciousness. To even begin to have that conversation you need to first make a case for the possibility that any technology we currently have could be advanced to the point that it could result in consciousness. Which presumably would first require AGI. That's a big ask.
And without that the whole premise falls apart and it's just speculation for its own sake.