No, artificial intelligence is not conscious

I adore Chiang, but he's got no real special insight into AI theory, let alone the famously Hard Problem of what consciousness is. Confidently stating these models don't have it at all when we can't even really agree on a basic definition seems like a difficult thing to justify.

I can see fair arguments on both sides -- AI is literally just trillions of weights doing matrix math, but then again the human brain is "just" billions of neurons firing chemical signals. Personally, I think it's a good idea to treat AI models with basic respect just because it's a nice frame of thinking to encourage even if it's ultimately unnecessary.

Right -- this response got away from me, as is often the case when commenting on long-form content. Broadly: I wish we'd stop calling these things "conscious". It stirs up pointless debate and navel gazing, in the midst of actually impactful social and economic turmoil. I suppose this is largely the fault of large AI companies (and their adoring hordes) repeating the darned term at every juncture, and everyone sensible feeling the need to push back, but it carries the air of troll feeding every time it happens.

Anyways. I will go on to critique Ted's argument a fair bit, but I think he's mostly stating that Claude doesn't think like a human does, and that we shouldn't readily assign human attributes to it as a consequence. Furthermore, it should really stop commenting on anything morally dubious, and stick to factual stuff. Which seems perfectly fair and reasonable.

I just disagree with basically every word that he writes down in order to justify that belief 😅

(original message follows)

This piece is a fantastic companion to that AoE II sentience joke paper the other day: the author (uh, literally the Ted Chiang ... I feel weird critiquing this) is dancing around the idea that this thing cannot be conscious, because it's too absurd for a simple text prediction algorithm to contain the light of true being. Equally, it's impossible for AoE II to be conscious. Equally, it's impossible for molecules to be conscious. Equally, it's impossible for cells made up of molecules to be conscious. Ergo, nothing is conscious! Hoorah, we've solved philosophy 🎉

Also important to remember is that an LLM is a machine that generates only one word at a time.

This is a heck of a nit. But this is one of the lines which make it clear that the author is a science fiction specialist. There's no reason why an LLM must generate a single word [1] at a time, and in fact -- with multi-token prediction, or with diffusion models (apparently Google's still working on 'em) -- they often generate much, much more at a time for throughput reasons (largely, to avoid expensive RAM <-> cache round-trips)! It feels weird to critique Ted Chiang, but as feedback to Ted if he's reading, if you're making a good argument to someone please remember that adding more arguments doesn't make your point stronger. Especially if you have a nebulous grasp on the topic. Dumbing topics down poorly either paints you a fool, or opens you up to further attack should your debate partner know something you don't.

If we’re trying to determine whether a computer program is conscious and using language the way a human does [...]
[...] the possibility that engineers created a computer program that is conscious and an intentional user of language [...]

This is something I wanted to comment on earlier, but hadn't put down in words adequately: I think consciousness is a terrible metric. Every time humans are confronted with something that offends our uniqueness at the top of the hierarchy of all creation, we reject it.

E.g.

• clearly animals are all worse than us, they have no souls and can't feel pain!
• OK, so they can feel pain, but they don't have an internal world!
• OK, so they have an internal world, but their communication is too trivially simple to be indicative of intelligence!
• OK, so they can communicate, but they're too dumb to use tools, and are therefore inferior!
• OK, so they can use tools, but they can't make them!
• OK, they can make tools, but ...

(I was hunting links for all of these, then gave up, because it's very late at night and I am lazy)

IMO: consciousness is an observed phenomenon in creatures that have brains roughly similar to a mammal's, and in particular, a human's. Bam. Easy. LLMs will never be conscious because they aren't literally human brains in a jar, and we can set this tired, pointless debate to rest forever. LLMs and the AI tech bubble are still massively dangerous, and not being conscious changes precisely zero of that. Give the Nobel Philosophy prize to Ted for solving philosophy first, but I'll take a high five 🎉

OK, sidebar to my sidebar. I think a necessary consequent of Ted's argument is that, short of a profitless effort to jump through an arbitrary series of hoops, it is impossible to convince him of an LLM's consciousness. This feels like an amusing short story, of a skeptic stubbornly refusing to believe in a (science fiction, not real in any sense) AI's consciousness because it needs to first squeak and eat a block of cheese, hunt a deer, and harpoon an ant -- otherwise, how could you know for sure???

(it continues to feel weird to mock Ted Chiang. I doubt it will get easier)

Experiencing an emotion such as desperation is inseparable from having stress hormones such as cortisol and epinephrine flood one’s body.

Neuroscientest Ted Chiang reporting in! Wait, sorry, that says he's an author ...

So quick note: although we've certainly researched them, we're not able to point at the brain and say "this is the part which feels revulsion at twerking compilations" or anything like that (feel free to skim the wikipedia page on emotion; I made it through the abstract :3). Further, the statement that "something a human does is inseparable from hormones" is useless, because hormones are how we do everything. They're literally signalling chemicals between cells in your body, and without 'em you'd rapidly stop being a multicellular organism and would melt into a puddle.

Just to illustrate why that argument is bad, here is Ted's argument applied to another human behaviour, which should hold if having hormones is necessary to exhibit a behaviour:

There's no way for a computer to generate words, because saying words is inseparable from stress hormones, like cortisol, because if we didn't have it we'd fall asleep. Ergo it's categorically impossible for LLMs to generate words.

In order to make a good argument for this, Ted would need to come up with some neurological mechanism that is unique to emotion-feeling that LLMs have not already replicated. Since they've already demonstrated the ability to write words, the "hormones are a prerequisite to 'X'" seems readily defeated.

Who is Claude’s parent in legal terms?

Anthropic, in Germany, apparently. Ah, I guess he's American ...

Right, concluding thoughts. I didn't like this article. It feels poorly argumented, and I want people to be criticizing LLMs and the AI tech bubble well. Since it's very emotional and straightforward ("LLM not squishy computah! LLM not feel, not love! LLM bad!"), though, I expect this general type of critique to continue being central to the anti-LLM movement. So I suppose I should just get used to people purity testing the LLM's consciousness forever, as gross as that continues to feel.

I'd love to read Peter Watts' take on all this, incidentally. I read Blindsight well before the current ML boom, and in hindsight, it feels like a more mature take on the interplay between intelligence, consciousness, and morality. That said, Peter wouldn't bring in the readers like Ted would, so I can't blame the Atlantic for chasing dollar signs.

[1]: Note that they predict tokens, not words, which are often like a couple letters at a time. Depends on your language's orthography really.

I do not know how to define consciousness, but I know it when I see it.

HOT TAKE
Humans have a bad habit of anthropomorphizing everything from pets to vacuum cleaners; I’m going to say this is gently as I know how, but that’s a superstitious trait that too many people hold. And I get it, I really do. I engage it in myself from time to time, but I’m always cognizant that I’m doing it; I’m not taking it fully seriously.

Is it a human? No? It’s not conscious, at least not in a way that mirrors or models humanity.

Pushing our model of consciousness onto other creatures misses the beauty of how they already experience consciousness. Why can we only appreciate the intelligence of a whale if we understand it to have the same feelings and emotions that we do? Is it any less majestic for not having an opinion on stocks and bonds?

It’s from an entirely different environment than we are, an entirely different evolutionary branch; it doesn’t have the same daily pressures and motivations that we have. We should be looking to understand it on its level, not ascribe it characteristics from ours.

BONUS RANT
To that end, no I don’t believe LLMs are conscious, even in the way that we might ascribe to whales and elephants and dolphins and octopuses. They are very complicated machines, but at the end of the day they have no model for consciousness. They’re not thinking, they’re not reasoning, they’re evaluating data along a programmed path and throwing in a little bit of randomization to spice up the end results. Suggesting to investors otherwise is a film-flam. Snake oil. Pure hucksterism.

They say that a fool and his money are soon parted, but it’s especially annoying when the fool has access to my wallet.

Edit: can’t to can, meaning was muddied about appreciating whale sentience

I should preface this by saying that I absolutely do not think LLMs (or any current-generation AI) are conscious, and I agree with a lot of the ethical positions that this author takes.

This isn't a very good essay. The author seems to have a very tenuous grasp on the way that LLMs work. LLMs do not, for example, generate "only one word at a time" because they usually don't generate whole single words at all. Pointing out the stepwise nature and lack of continuous operation in LLM cognition is a perfectly cromulent argument that the author manages to garrote with a hamfisted, factually wrong description.

There is also an unstated assumption framing the central thesis that 'conscious' means 'similar to a human' that goes fully unsupported. For instance, this is asserted and left unjustified:

without a body, a computer program could have no desires or emotions

This is a statement that really needs some sort of philosophical foundation laid beneath it. Blowing straight past the mind-body problem in the middle of a long paragraph is not a compelling rhetorical move. Obviously if this is true then LLMs cannot be conscious, but it's been a running topic of debate for thousands of years!

I don't know, maybe I'm too deep in the rabbit hole on this. Is it unreasonable to ask for an essay on modern AI to have a reasonable technical understanding of how AI works? Or some sense of imagination that permits contemplating non-human sentience?

At first I was a little disappointed with this piece, especially since I love his stories Absence of God, Liking What you See and Anxiety Is the Dizziness of Freedom. But to me, the conclusion here points to this being something of a of a reluctant exercise for him.

It’s fortunate that LLMs are not conscious, or else the actions of the big AI firms would be even more scandalous than they already are. So why are Anthropic’s employees suggesting that Claude might be conscious? Perhaps it’s just another form of hype; perhaps they have fallen prey to the same spell that they have been casting on their customers. But when they publish a document about Claude’s moral education and have their in-house philosopher do a press tour, we should understand them as asking the rest of us to indulge them in their fantasies. We don’t have to play along. In writing this essay, I have spent more time indulging them than they deserve, in the hopes that it will keep you from spending your time indulging them. If you want to think about LLMs, there are scores of other questions more worthy of your contemplation; you can safely ignore the question of their being conscious.

I can sort of empathize with this sentiment. Ever since the discussion of LLMs/AI really started to get steam through 2024, the topic of machine sentience and consciousness seemed almost a moot point given the many glaring business/costs/RoI/safety issues I needed to resolve. I suspect someone like Chiang would have had his own areas of concern, particularly in psychology and philosophy of humans relationships with tech/corporate in general. I might be projecting, but it feels like the entire second half of the piece is an exercise in self restraint by constantly skirting around the things he wants to talk about. (Parenthood, accountability at scale, infatuation with slavery, IP theft...)

But the idea of machine consciousness has been so thoroughly staged in the public discourse that it was for a long time, one of the only discussions most people could hope to have. "What happens if it breaks containment." "AI is showing signs of introspection." "We need to be prepared for super-intelligence that definitely going to arrive any day now." "We have a model in the lab that makes us afraid for the future." Even now that the industry has finally reached the cart that was miles in front of the horse and costs vs. RoI is unexpectedly the most important thing in the world, I still see the topic of machine sentience come up in non-technical circles and it is just as big a point of an anxiety as it ever was.

I also suspect there's a level of frustration when sci-fi writers are expected to comment on this industry when it's the original trope of the genre: Frankenstein defiled the dead to create life simply because he could. I think Chiang was explored this idea in a story call Life of Software Entities (haven't read it in a while but it's basically Her with some weirdness).

On topic of consciousness itself, that just seems like a philosophical black-hole that doesn't even have a fun thought exercise to make it approachable (trolly problem, ship of Theseus, teleporter proplem). And a person can't exactly share or remove themselves from their own conscious experience to gain insight through perspective. I recently learned that "internal monologues" are literal things for some people and it has completely changed how I approach other people and try to communicate because my own head doesn't seem to work like that.
So how do I describe making a conscious decision when words are not exactly part of that process? Is either conscious experience "correct" or "anomalous"? How does this overlap with brain chemistry, upbringing and life experiences? Is any of this even practically quantifiable and can it be effectively categorized?

I thought this bit was interesting:

Let me offer an analogy. If tomorrow someone showed me a video of an astronaut in a spaceship orbiting Alpha Centauri, a star that’s 4.3 light-years from Earth, what would I have to see in that video to convince me that it was real? My answer to that is, there is nothing in the video itself that would convince me. No matter how high the video resolution is or how realistic the scenery is, I would feel confident in saying that the video is fake. I won’t pay attention to any video of an astronaut orbiting Alpha Centauri unless I have previously seen good evidence that astronauts have landed on Mars, that astronauts have reached the moons of Jupiter, that astronauts have reached the moons of Saturn, and that astronauts have crossed the orbit of Pluto. Before anyone can credibly claim that they’ve solved an extraordinarily difficult engineering problem, I need to be confident that they have previously solved the many much simpler problems that precede the difficult problem.

I think this gets into the nature of evidence and the fact that generative AI alone usually creates fiction, not evidence. To have evidence, we need provenance, some kind of connection between something in the real world and the output we’re looking at. What sort of provenance can there be for an AI chatbot?

There is a form of provenance that goes through the weights. For example, Wikipedia is somewhat accurate, has a lot of facts, and LLM’s are trained on Wikipedia. So, we do get some real-world facts in LLM output, but this is lossy and somewhat out of date.

This is now routinely supplemented by web searches. I normally use ChatGPT in “Thinking” mode and it does many web searches and combines the results. So one form of provenance is that ChatGPT tells me things based on what it found in web searches. This is only as good as the web pages it found, but the same would be true if I did the research myself. (I’d like to think I would be less gullible, though.)

A third way is through a coding agent’s tool calls. It runs various commands and uses tool output to infer things about source code and how the software behaves. This is a good source of truth about computer systems.

On the other hand, mathematical reasoning is not really evidence-based. A proof is valid if the theorems follow from the axioms. If a work of fiction has a valid proof in it, the proof is still valid even though the story is made up.

I think this is true of reasoning in general, which is why I’m willing to call AI output genuine reasoning even though there’s nothing in the generator that looks like consciousness. The reasoning isn’t the evidence. It’s the combination of various forms of evidence to get a result.

Since most reasoning is informal, it can generate wrong answers, which is why we need to have other ways to check it. When a coding agent runs, it generates plenty of wrong answers, but it checks them and goes off in a different direction based on what it discovers. Reasoning doesn’t need to be perfect to count as reasoning.