How not to 'debunk' AI sentience

A recent TIME article announces “No, Today’s AI Isn’t Sentient. Here’s How We Know”. In fact, we know no such thing, nor does the article present an argument or evidence that could deliver such knowledge. Now, the article does clearly rebut a poor argument for AI sentience based on a naive interpretation of language model outputs, and in this I agree with it. But along the way, the article conflates sentience, human-like sentience, intelligence, and risks from misaligned AI. Untangling these conflations sheds light on some key principles for thinking clearly about potential AI sentience.

The article, by AI researcher Fei-Fei Li and logician John Etchemendy, begins by noting increased public and intellectual interest in risks from AI, and in the possibility AI sentience. The authors define sentience, sensibly and in keeping with a common usage, as “the ability to have subjective experiences—to feel what it’s like, say, to experience hunger, to taste an apple, or to see red.” (“Sentience” in the sense of the capacity for subjective experience, is also often called “phenomenal consciousness” or “consciousness”. I will use “sentience” and “consciousness” interchangeably in what follows.)

But in short order, the article proceeds to conflate several crucial issues that are best kept quite separate.

1. Conflating sentient AI and powerful, misaligned AI

Li and Etchemendy present the case for AI risk as if it is essentially the same as the case for sentient AI systems. “The most prominent position” in recent AI discussions, they write, “is that the emergence of ‘sentient AI’ could be extremely dangerous for human-kind, possibly bringing about an ‘extinction-level’ or ‘existential” crisis’. After all, a sentient AI might develop its own hopes and desires, with no guarantee they wouldn’t clash with ours.” 

The “prominent position” that advanced AI systems could be dangerous does not, in fact, look anything like this.

The prominent arguments for AI risk—every one that I am aware of, whether popular or academic—either studiously avoid discussion of AI consciousness / sentience or actively separate AI sentience and AI risk. And rightly so. We don’t know enough about sentience to know how, if at all, it relates to general intelligence and risks from misalignment. And we certainly do not know, as Li and Etchemendy assert without argument, that “sentience is a crucial step on the road to general intelligence.” 

As a result, many prominent explications of AI risk go out of their way to clarify that risks from advanced AI do not depend on AI systems gaining sentience/consciousness:

• Bales, D’Alessandro, Kirk-Giannini (2024): The argument for risks from AI “doesn't rely on a unified notion of general intelligence. Nor does it assume that AIs will become conscious, acquire personhood, or possess humanlike beliefs and desires. Instead, what matters is that the systems are highly capable at various tasks that are useful for gaining power.”

• Ngo, Chan, and Mindermann (2024) mentions “consciousness” once, to point out that risks from “situationally aware” AI do not require consciousness.

• Chalmers (2010): “for the purposes of the argument, intelligence is to be measured wholly in terms of behavior and behavioral dispositions…If there are systems that produce apparently superintelligent outputs, then whether or not these systems are truly conscious or intelligent, they will have a transformative impact on the rest of the world.”

• Butlin et al. (2023): “arguments that AI could pose an existential risk to humanity do not assume consciousness. A typical argument for this conclusion relies on the premises that (i) we will build AI systems that are very highly capable of making and executing plans to achieve goals and (ii) if we give these systems goals that are not well chosen then the methods that they find to pursue them may be extremely harmful (see e.g. Hilton 2022). Neither these premises nor the ways in which they are typically elaborated and defended rely on AI systems being conscious.”

So from the start, the framing of the article to conflate sentience, intelligence, and AI risk—without argument, while misattributing this conflation to AI risk proponents—sets the article off on the wrong track. As we’ll see, this conflation leads them to dismiss AI risk and sentience together in one single dubious inference.

2. Rebutting a pro-sentience argument based on LLM self-reports 

The authors then consider an argument in favor of AI sentience: “AI is sentient because it reports subjective experience”. This argument, as they present it, claims that we get exactly the same evidence for sentience when a large language model (LLM) outputs “I feel hungry”, as we do when a human says “I feel hungry”:

I believe that you, as a person, are conscious when you say ‘I have the subjective experience of feeling happy after a good meal.’ I, as a person, actually have no direct evidence of your subjective experience. But since you communicated that, I take it at face value that indeed you have the subjective experience and so are conscious. 

Now, let’s apply the same ‘rule’ to LLMs. Just like any human, I don’t have access to an LLM’s internal states. But I can query its subjective experiences. I can ask ‘are you feeling hungry?’ It can actually tell me yes or no….I have no reason to believe it’s not conscious or not aware of its own subjective experiences, just like I have no reason to believe that you are not conscious. My evidence is exactly the same in both cases.

The authors then reject this argument. And I agree wholeheartedly with their rejection. They argue, correctly, that the evidence is not the same in the two cases, because there are “profound difference between how humans generate sequences of words and how an LLM generates those same sequences”. And at least for reported states like hunger, we have far more corroborating evidence for humans (i.e., they have stomachs).

My main issue with this section is that this argument is presented as being “quite representative of those in the ‘sentient AI’ camp”. Of course, this may be true of the people that the authors have been speaking with. And since the TIME article is not an academic article and makes no claim to being comprehensive, it is perfectly fine for the authors to discuss only the arguments that are most salient to them.

by GPT-4o

But many of the best discussions of potential AI sentience do recognize that current LLM outputs are (taken at face value) a poor guide to consciousness. The discussion does not begin and end with that single source of putative evidence, just as the discussion of parrot sentience does not begin and end with the fact that parrot “utterances” are spurious. That is why a recent paper on AI consciousness (COI: I’m an author) is skeptical of purely behavioral evidence for sentience and looks for architectural and computational indicators drawn from scientific theories of consciousness.

3. Assuming that sentient AI systems must have human-like experiences

The authors then rule out LLM sentience entirely, by restricting sentience to human-like experiences. This restriction leads them to dismiss AI sentience far too easily— and given the earlier conflation of sentience and AI risk, to dismiss AI risk as well.

“All sensations—hunger, feeling pain, seeing red, falling in love—are the result of physiological states that an LLM simply doesn’t have. Consequently we know that an LLM cannot have subjective experiences of those states. In other words, it cannot be sentient.”

The problem is that this last sentence is that “it cannot be sentient” does a lot more than recap their preceding points “in other words”.

First, it is a substantive claim, not merely a matter of definition, to restrict sentience in general to “sensations” that are “the result of physiological states”. It’s an open question how and whether physical embodiment is needed for sentience, and whether non-bodily states could be conscious. Second, as the next section discusses, plenty of AI systems have bodies. Third, perceptual states like “seeing red” presumably depend on the processing of visual information, which multimodal LLMs obviously do.

More generally, sentience need not only include human-like sensations like hunger, pain, and emotion. We should be wary of thinking that AI systems (or animals) would have to have the same kinds of sensations as we do in order to be sentient, and of only looking for human-like sensations in AI systems.

This distinction between sentience in general, and particular human experiences, will be familiar from Thomas Nagel’s seminal paper. Nagel points out that bats are plausibly conscious/sentient but would have experiences that humans do not. By the same token, if AI systems have sentience, they could be (and maybe would tend to be) conscious of different things than we are. One cannot dismiss the entire question of AI sentience—or AI risk!—just because LLMs cannot feel hungry.

4. Conflating LLMs with AI systems in general

In the piece, Li and Etchemendy argue that large language models cannot be sentient. But the claim made in the title of the piece, and repeatedly in the text, is that AI is not sentient. But there are many more AI systems besides LLMs. And the piece contains no discussion at all about the variety of advanced AI systems being developed today, many of which incorporate LLMs but augment them in various ways. Many of these AI systems do not obviously fail to meet the requirements that Li and Etchemendy place on sentience, like having a body and sensing that body. Given the article’s emphasis on embodiment, it is especially strange that it does not mention any of the sophisticated robotics systems that incorporate LLMs.

by GPT-4o

In a recent and prominent NeurIPS talk (and subsequent paper), David Chalmers argues that these augmented or extended AI systems, which he calls LLM+, overcome many common objections to LLM consciousness. As a result, “the case against consciousness in current LLM systems is much stronger than the case against consciousness in future LLM+ systems.” 

Given the increasing prominence of systems that go beyond pure-text language modeling, in robotics and elsewhere, it is irresponsible to dismiss the entire issue of AI sentience (or of AI risk) based solely on LLMs. The authors note that their article is “not meant to be complete or comprehensive”. But that does not excuse narrowing the scope in this way. The article’s approach is akin to arguing that octopuses are not sentient, then declaring “No, Today’s Animals Are Not Sentient. Here’s How We Know”. As AI systems continue to advance and proliferate at breakneck speeds, it is important that we not restrict our attention to large language models.