Brain as computer: why this metaphor can be misleading

In this article you will see:

• What Anil Seth, Ned Block and Giulio Tononi argue in Nature
• Why the cerebellum shows that intelligence is not consciousness
• What the report by 19 specialists concluded about LLMs
• Why language models seem conscious without being so

What Seth, Block and Tononi say

Anil Seth, Ned Block and Giulio Tononi have spent decades researching consciousness from perspectives of experimental neuroscience, theoretical neurobiology and philosophy of mind. Their commentary responds to voices like biologist Richard Dawkins and AI pioneer Geoffrey Hinton, who consider AI systems possibly already conscious.

The authors also cite a study that Anthropic published this year. The company reported internal processing patterns in Claude that resemble properties of consciously accessible information processing. Anthropic does not see this as evidence of subjective experience.

The cerebellum is intelligent but not conscious

The central evidence of the authors is the human cerebellum. It contains almost four fifths of all neurons in the brain and plans movements with precision, speed and learning ability. None of this happens consciously. If the cerebellum is removed, consciousness is not directly affected.

The authors trace the confusion between intelligence and consciousness to two psychological tendencies. People see the world through their own lens and attribute human qualities to things. Language was considered something exclusively human for a long time. But language models are trained on large volumes of human text that deal with thoughts and feelings. As the authors say: LLMs are mirrors that reflect an image of our collective digitalized past.

For comparison, Seth, Block and Tononi point to AlphaFold, from Google DeepMind. It is also a powerful neural network on silicon hardware, but almost nobody considers it conscious. The reason, according to them, is simply that AlphaFold predicts protein structures instead of talking to us.

This debate recalls the recent discussion about computation and intelligence, as we showed in the post about Rich Sutton reinforces that compute alone does not create intelligence. The same Nous Research that reeducated Qwen for Western values, as we showed in the post about Arete-27B: Nous reeducates Qwen for Western values, also discusses how bias and behavior manifest in language models.

What the Butlin et al. report concluded

To understand the scope of the commentary, it is worth looking at the report "Consciousness in Artificial Intelligence", by Patrick Butlin, Robert Long and 17 other specialists, including AI researcher Yoshua Bengio. The team drew on neuroscientific theories about how consciousness arises in the human brain and derived a list of 14 indicators. The more indicators an AI system meets, the stronger the case that it could be conscious.

The researchers then evaluated several current systems, including language models like GPT-4. The report concludes that none of the systems examined is a strong candidate for consciousness. On this point, it aligns with Seth, Block and Tononi. For language models, Butlin and colleagues found only weak signs that the models meet the indicators of one of the most prominent theories of consciousness.

At the same time, the report sees obvious technical barriers to building future systems that meet more indicators. Whether such systems would actually be conscious remains open. The authors softened an earlier stronger formulation on this point.

There is also a fundamental premise. The team assumed that consciousness depends only on the type of information processing, not on happening in nerve cells or computer chips. Experts call this computational functionalism. The authors adopted this premise explicitly for pragmatic reasons, because it is what makes it possible to examine AI systems for consciousness. The report itself acknowledged that its indicators would be less reliable if consciousness were tied to properties of living organisms. It cited Anil Seth, among others, as a defender of this opposing view. That is where the Nature commentary connects.

Why LLMs seem conscious without being so

The authors argue that consciousness is what disappears during dreamless sleep or general anesthesia and returns on waking or dreaming. Inspired by philosopher Thomas Nagel, they describe it like this: conscious experiences feel like something. Intelligence, on the other hand, means performing functions. One does not follow from the other.

The Butlin et al. report rejected behavioral tests for AI consciousness, because AI systems can be trained to imitate human behavior even while working completely differently on the inside. They also warned that language models will likely lead many people to believe they are talking to a conscious counterpart. As an example, they cited former Google engineer Blake Lemoine, who believed the chatbot LaMDA was sentient.