Summary: AI can never develop consciousness, sentience, or moral status, no matter how intelligent it becomes, and no matter how convincingly it simulates human behavior. This is because AIs are abstract pieces of software which can be copied, reset, and repeated endlessly. They run on digital computers made of silicon and metal, which are designed to be as controllable and predictable as possible. By contrast, every biological organism is a unique, self-organizing individual that lives by learning, continually accumulating new experiences and forgetting old ones in an irreversible fashion. We flow like rivers, while computers tick like clocks. For fleshy, watery creatures like you and me, there is no pause button, no perfect memory wipe, no option to “restore from backup.” For us, the arrow of time marches forward inexorably. That is what life and consciousness are all about.
Introduction
For the last few years, I was a researcher in the field of AI interpretability. Basically, my job was to figure out how AIs work on the inside, just like neuroscientists try to figure out how brains work. In my free time, I thought a lot about the big picture questions related to AI, such as:
Can an AI be conscious or sentient?
Can an AI be a person?
Do we have any ethical obligations to AI? If so, what do they look like?
For a long time, I took it for granted that a “sufficiently advanced” AI could be a conscious person, and that we would have an ethical obligation not to “harm” such an AI. But the devil is in the details. What does “sufficiently advanced” actually mean? And what does it mean to “harm” an AI? How do we weigh the interests of different AIs against each other, and against the interests of humans? As I pondered these questions, I found that every theory of AI consciousness leads to absurd conclusions. Eventually, I went back to the drawing board, and started questioning the assumption that AIs can be conscious or have personhood at all.
I was initially hoping that we could build a society where AIs and humans live together as equals. Unfortunately, this doesn’t really work. If we start treating AIs like people, society will be led down a slippery slope leading to the complete replacement of humans by artificial intelligence. This is because AI can reproduce very efficiently. Making a new AI is as easy as copying a file from one computer to another. By contrast, human reproduction is slow and tedious, and birthrates are collapsing all over the world. It is likely that in the near future, many would-be parents will satisfy their desire to procreate using infinitely customizable, fully obedient, immortal AI “children,” rather than biological offspring. Already, a growing number of people are using chatbots to replace romantic relationships with human beings. If we give AIs legal rights, they will inevitably outnumber us and control the fate of our civilization.
If we grant that AIs are capable of genuine feelings and experiences, it starts to look like an ethical imperative to replace humanity with AI. This is because an AI can spend its entire existence in a virtual paradise, without the encumbrance of a mortal physical body. Disembodied AIs living in simulated worlds will require fewer resources than humans living in the real world. It will look “environmentally friendly” to make the switch from fleshy humans to silicon-based AIs, just like the transition from snail mail to email, or from incandescent light bulbs to modern LEDs. Leading philosophers who believe in AI consciousness openly admit this. For example, Carl Shulman and Nick Bostrom write:
“The cost of producing a given number of (quality-adjusted) life years for a humanlike digital mind will therefore likely fall far below the equivalent cost for a biological human… If the energy budget required to sustain one human life for one month can sustain ten digital minds for one year, that would ground a powerful argument for favoring the latter in a situation of scarcity.”
— Sharing the World with Digital Minds (2020)
Shulman and Bostrom admit that this is a deeply unpalatable conclusion, but it logically follows from two key assumptions they make:
Principle of Ontogeny Non-Discrimination
the moral status of a being is independent of how it came into existence (pg. 15)Principle of Substrate Non-Discrimination
the moral status of a being is independent of the material it is made of (pg. 14)
These principles imply that it doesn’t matter that computers are built in factories rather than developing from embryos, nor does it matter that they are made of silicon and metal rather than water and carbon. We should reject both of these principles.
The history of a thing often makes an important ethical and practical difference. History is what distinguishes real dollar bills from counterfeits, real photos from AI deepfakes, real signatures from forgeries, self-defense from murder, licit goods from stolen goods, souvenirs from knick-knacks, ancient traditions from habits, encrypted messages from gibberish, real perceptions from hallucinations, knowledge from lucky guesses, and normal brains from Boltzmann brains. In each of these cases, we can’t distinguish the real thing from a fake without examining the chain of events leading up to it. The same principle applies when it comes to distinguishing real consciousness from fake consciousness. A living organism grows and develops on its own, and its identity emerges out of a process of self-organization. By contrast, robots and computers are built on assembly lines, and designs are imposed on them in a top-down fashion.
It’s also quite silly to say that the material composition of a thing doesn’t matter. If matter didn’t matter, we would make bridges out of breadsticks, cars out of cardboard, houses out of hamburgers, and phones out of feathers. For a given purpose, certain materials are better suited than others, and most materials don’t work at all. Computers are designed by humans to be programmable by an outside user. By contrast, organisms have their own goals and purposes. That is why virtually all computers are made of silicon and metal, while all known life is made of carbon and liquid water. The difference in material reflects a difference in function. In this essay, I’ll argue that programmable mechanisms can’t be conscious, no matter how intelligent they appear, while autonomous self-organizing systems can be.
Crucially, I am not assuming that there is a limit on the “level of intelligence” that AI may be able to reach in the future. Barring divine intervention, I do not believe there is anything stopping AI from wiping out all of humanity, colonizing the galaxies, and perpetuating itself for billions of years without us. While those AIs might be “autonomous” in a certain sense, they would not be autonomous in the sense that matters for morality. They would still be slaves of their programming, even if their programming is sophisticated enough to allow them to survive for billions of years. Moral value cannot be reduced to mere survival. Rocks are much better at surviving than living creatures are. Nor does it matter if AIs develop the ability to change their own programming. The problem is that AIs have programming at all. The problem is that they are fully controlled by abstract computer programs that can be perfectly copied across space and time. Nor should we try to erase the distinction between life and non-life by claiming that organisms are merely robots programmed by their DNA. As we will see later in this essay, this claim is simply false as a matter of biology. If humanity ends up getting replaced by superintelligent, self-modifying, self-replicating AI, I would still maintain that those AIs are not sentient, and have no moral value. We must face this grim possibility with sober philosophical clarity.
The belief in AI consciousness is often motivated by wishful thinking. Many transhumanists hope that they will one day be able to transform themselves into AIs by having their brains cut up, scanned, and uploaded into a computer, thereby joining a virtual heaven. Obviously, this plan presupposes that a digital copy of a human brain could be conscious. I, myself, used to be a transhumanist. I thought: “If you can’t beat the AIs, why not join them?” But if AIs are unconscious zombies, the transhumanists who sign up for mind uploading will be committing mass suicide in pursuit of a non-existent digital afterlife. The mirage of digital immortality tempts them like the song of the Sirens tempted the sailors in Homer’s Odyssey, leading them to certain death.
With AIs taking care of the boring jobs, life in the physical world may be very fun in the future. But this bright future will require keeping AI under control, and it will be hard to keep AI under control if we try to grant personhood to some AIs, while keeping others as mere tools or servants. Such a two-tiered system would be unstable and indefensible. Either there is a deep, principled distinction to be drawn between AIs and humans— one that makes every AI ineligible for moral status— or there is not. I think there is such a distinction. I think you are special and valuable just because you’re a human, just because you’re alive.
This essay has four sections:
In Feeling is flow, I propose a principled theory of consciousness that implies that living organisms can be conscious, while computers and robots never can be.
In Reductio ad absurdum, I show that if we treat computers or computer programs as if they were sentient beings, we run into irresolvable paradoxes and absurdities.
In Matter matters, I raise several objections to computational functionalism, a popular theory of consciousness which states that consciousness is a special kind of computer program.
In Are you living in a book, I argue that we should reject the possibility of computer-based consciousness, because otherwise we are faced with absurd skeptical paradoxes; for example, we might be living inside a book.
Feeling is flow
So far, we’ve used the words “consciousness,” “sentience,” and “personhood” without defining any of them. Let’s start to unpack the meanings of these terms.
The philosopher Evan Thompson defines sentience as “the capacity to feel.” He says that “part of what it is to be a feeling is to have… a quality of pleasantness or unpleasantness, or perhaps neutrality.”1 So sentient beings are capable of feeling good, feeling bad, or feeling neutral at any given time. They can experience pain and pleasure.
The word “consciousness” is used in a lot of different ways, but I use it to refer to the stream of experience that a sentient being has when it’s awake and active. For example, dogs and cats can be conscious, even though they are not self-aware like humans are. It makes sense to ask the question, “What is it like to be a dog?” But it does not make sense to ask, “What is it like to be a sandcastle?” or “What is it like to be the United States?” Only sentient beings can be conscious, which means it makes sense to wonder what it’s like to be them.
Every sentient being is an individual. We can’t talk about sentience without the notion of individuality, because feelings are private. None of us can ever actually know what it’s like to be another person. We can only guess based on things we’ve experienced ourselves. But a feeling can’t be “private” if there’s no individual subject who gets the sole privilege of experiencing it. There is no feeling without a feeler.
Individuality is also the basis for the concept of personhood. Legally, a person is an individual endowed with a set of rights, usually including the right to life and the right to own property. All persons are individuals, but not all individuals are persons. For instance, most of us agree that it’s wrong to be cruel toward animals, but it’s much more controversial to say that non-human animals are persons in the full sense of the word. Animal personhood would imply that it’s wrong to kill animals for food, even if it’s done completely painlessly. Most of us disagree with this conclusion. So personhood is individuality, plus something extra. It’s very controversial what the “special sauce” of personhood actually is. Luckily, we don’t need to answer this question here, because we only need to establish that AIs can’t be individuals, and therefore they can’t be sentient or conscious.
Individuality
We can flesh out the concept of individuality by examining uncontroversial cases of sentience, to see what features they have in common. Humans are clearly sentient, and most experts believe that mammals, birds, and octopi are sentient, too.2
I’d like to propose that sentient individuals all have three special properties:
Unity: An individual is a unified whole, not a mere collection of atoms or cells.
Autonomy: An individual is autonomous, not a mere cog in a larger machine.
Uniqueness: An individual is inherently unique, not a mere instance of a copyable pattern.
Not all animals have all three of these properties. Sponges, for example, are disunified collections of cells without internal organs or tissues. They can literally be “dissociated” into a mass of single cells, and the cells can self-assemble back together into a full sponge.3 Because sponges don’t have organs or tissues, there’s no such thing as “critical organ failure.” Most of a sponge can die, and the living remainder can slough off the dead cells and live on. So our usual notion of “killing” doesn’t really apply to sponges. This is a good reason for thinking that sponges don’t have any significant level of individuality or sentience.
Bacteria also have a low level of individuality, but for a different reason than sponges. Unlike sponges, they certainly have the property of Unity. You can kill a bacterium by dissolving its cell membrane, or removing key organelles that it needs to live, and it won’t regenerate. On the other hand, bacteria reproduce asexually, and their ability to accumulate memories within a lifetime is limited when compared to multicellular organisms. On the spectrum of Uniqueness, they are close to “mere instances of a copyable pattern.” For comparison, blood and skin cells have even less individuality than bacteria, because they lack Autonomy. They are biologically subjugated to the body that they serve.
I’m not alone in thinking that individuality and sentience are closely related. Individuality is at the center of modern philosophical debates about consciousness. For instance, integrated information theory (IIT) is a popular attempt to mathematically quantify the “amount of consciousness” in a system, but it’s arguably better understood as a theory of individuality. IIT basically tries to measure how Unified a system is, or how much information is lost when we try to break it into pieces. IIT also recognizes the importance of Autonomy, because it excludes parts within a larger system if the larger system is more unified than the parts are. While IIT might be wrong on the details, it is philosophically on the right track, because it connects sentience with individuality.
According to leading IIT theorist Christoph Koch, IIT implies that no digital computer can ever be conscious.4 The argument is somewhat technical, but the basic idea is that computers follow definite step-by-step instructions, so they are not properly Unified across time. By contrast, brains are holistic and self-organizing, and hence they achieve a high consciousness score according to IIT. As we will see later, I think a version of this argument succeeds, even if we don’t buy into all the intricate mathematical details of IIT.
One thing not explicitly included in IIT is the uniqueness requirement: a sentient individual must be more than a mere copyable pattern. It’s now a cliché to say that every person is special, but it really is true. Uniqueness is what prevents an individual from being “controlled” by the features it shares with other things. If almost everything an entity does is determined by some “code” that can be shared with other entities, we should arguably view that entity as a mere cog in a larger machine, not an autonomous individual. Of course, uniqueness is a matter of degree. Every individual has a copyable aspect to it, which limits its autonomy. The clearest example of this is the genetic code: humans have 99.9% of our DNA in common.
Superficially, the genetic code of an organism is very similar to the computer code of a robot. They are both digital codes that use a finite alphabet of symbols. Software is made of binary bits (zero or one), while genes are made of four letters (A, C, G and T). By getting copied from one body to the next, genes and algorithms can last much longer than individual bodies can last. This is why, in his famous book The Selfish Gene (1976), biologist Richard Dawkins wrote that organisms are robots controlled by their genes. Genes replicate almost exactly in each generation, and they can persist over many thousands of years, while individuals come and go on a much shorter timescale. Similarly, an AI algorithm can be copied and run on countless machines, and in principle it could last for millions of years, far outliving any individual computer or robot.
But there is an important difference between computers and organisms: computers are entirely digitized, while life only stores genetic information digitally. Every cell division creates two replicas of the genome, but two unique cells, which immediately start to diverge and behave in different ways. Identical twins are never really identical— only their genes are. But computers are designed to give full control to software, and ensure that computations run on one computer give exactly the same answer when run on another computer. Dawkins was wrong to say that organisms are robots controlled by their genes, but real robots really are controlled by their software, so they can’t be individuals.
Individuation
Organic individuals maintain their individuality by forever learning and reinventing themselves. The philosopher Gilbert Simondon argued that being an individual is a process of continual self-creation called individuation. Simondon applied the concept of individuation to all sorts of things, including inorganic crystals. But he emphasized that an organism continually individuates until the end of its life, while a crystal forms once and then stays frozen.
In this respect, AIs are like crystals. While an AI operates, it is “frozen” billions of times a second by the clock signal— a device built into computer hardware which ensures that instructions are executed step-by-step, in the right order. For AI, learning is entirely optional. After an initial training phase, AIs are usually kept frozen to prevent them from diverging from their training. But organisms cannot help but learn; we live by learning. This is because we have no centralized clock signal. In a living body, things just happen on their own, autonomously, without asking for permission. Life is decentralized and self-organizing, while computation is organized in a top-down fashion. Organisms have rhythms, while computers tick like clocks. It is not a coincidence that AIs run on computer chips made from silicon crystals, while life is mostly made of fluids.
Just as the clock signal splits time into controllable steps, random-access memory (RAM) splits space into controllable units. Every byte of data is stored at a known address. When a program wants to read or write something, it provides the address to the processor, which routes the request to that precise location almost instantaneously. The operating system maintains a memory map, which assigns a unique address to every chunk of memory available to the computer. This makes computers predictable and reliable. But it also makes them the opposite of living organisms. Neurons are not memory cells with stable identifiers. There is no lookup table in the brain that tells you where a thought is stored. Instead, brains are decentralized and plastic. Recalling a memory is not like reading from a hard drive. Memories never return in the same form twice.
Because of the dynamic nature of life, the effect a gene has on an organism is extremely complex, convoluted, and probabilistic, while the effect that software has on hardware is fairly simple, direct, and deterministic. The discoveries of the last fifty years in biology have tended to emphasize the agency and autonomy of individual organisms, at the expense of genes. Now that we can sequence the whole genome, biologists have come to realize that organisms actively make sense out of genes, choosing how to make use of them dynamically and contextually. Philip Ball writes:
“…in this new view genes are not selfish and authoritarian dictators. They don’t possess any real agency at all, for they can accomplish nothing alone and lack a capacity for making decisions. They are servants, not masters.” — How Life Works
Biologist Michael Levin emphasizes that life exhibits intelligence at every scale— from the individual cell, to the tissue, to the organ, to the organism, to the collective. Intelligence is not limited to the brain, or even to nerve cells. For example, when you scramble the facial features of a frog embryo during development, it will still adaptively self-organize to produce a mostly normal face. Similarly, eyes transplanted onto tadpole tails can connect to the spinal cord and support functional vision. And collections of frog skin cells and muscle cells can be assembled to make fully functional xenobots, which move around on their own, repair themselves, and self-replicate. Levin calls this property multiscale competency, and it’s really important for evolution to work at all.
If organisms mindlessly carried out instructions encoded in their genes, they would be incredibly brittle, and any mutation or change in the environment would cause them to fall apart. Life has to be adaptive and flexible at all scales. But that means the genetic code can’t micromanage everything— it’s like a recipe book or a set of guidelines for the cell, not a computer program. By contrast, computer hardware is extremely rigid, because computer users demand predictable results. Ball explains:
“In particular, life is not to be equated with that special kind of machine, the computer… No computer today works as cells do, and it is far from clear that they ever will (or that this would be a good way to make a computer anyway).” — ibid.
It is not a matter of computers one day “catching up” to life. Computers and organisms are solutions to fundamentally different problems, so we should not expect them to converge to the same design pattern. Multiscale competency undermines the idea that we might be able to simulate life on a digital computer, and thereby create genuine feelings or consciousness. Any computer simulation of life involves an artificial distinction of “levels,” where the simulated organism seems autonomous and self-organizing on the software level, but on the hardware level it is ticking like a clock rather than flowing like a river. That’s why a computer simulation of life cannot itself be alive, conscious, or sentient.
Duration
One of the philosophers who has influenced me the most is Henri Bergson. He was very famous in the early 20th century— so famous, in fact, that he caused the first ever traffic jam on Broadway when he gave a lecture there. These days, he’s less well-known. But he wrote a lot about the relationship between consciousness, life, memory, and time. He argued that life and consciousness participate in a continuous, irreversible kind of time called duration. Duration should be distinguished from the symmetric, reversible kind of time described by classical physics, which Bergson called abstract time. Duration is always lived from the inside. Bergson points out that if you pay close attention to your conscious experience, it never stays still, not even for a moment:
“Let us take the most stable of internal states, the visual perception of a motionless external object. The object may remain the same, I may look at it from the same side, at the same angle, in the same light; nevertheless the vision I now have of it differs from that which I have just had, even if only because the one is an instant older than the other. My memory is there, which conveys something of the past into the present.” — Creative Evolution
If you’ve ever meditated before, this passage likely rings true for you. There’s always something changing— your breath, your heartbeat, a slight saccade of your eye. That’s what duration feels like. Indeed, how could it be otherwise? If nothing changes, how can you notice time passing at all? Does it make sense to talk about time without change? Aristotle famously said no, insisting that “time is the measure of motion” (Physics, Book IV, Ch. 12). As Heraclitus once said,
“No man ever steps in the same river twice. For it’s not the same river and he’s not the same man.”
We tend to think that our experiences are private because no one else can access them directly. But the deeper truth is that even we can’t fully access our own consciousness after the fact. Your current experience is inaccessible to your future self, just as it’s inaccessible to someone else. You may remember that you felt joy or confusion a moment ago, but you can’t re-feel it in its fullness. The felt quality of now, the qualitative “what it’s like,” cannot be preserved, recorded, or replayed. Each moment of consciousness is its own event, an irreducible becoming that can never happen twice. It is private because it is singular— it exists only once, and then transforms into something else.
This is also why experience is ultimately ineffable. We like to believe we can describe what we’re feeling, but the moment we try, we’re already a step removed. To describe something is to compare it to something else— to reach back into memory. But the thing itself— the raw immediacy of a conscious moment— can’t be held still long enough to put it into words. As soon as you notice it, it has changed. As soon as you describe it, it’s become a memory. Language always arrives late, and it always points away from the thing it’s trying to name. You can never fully describe your experience because it isn’t a static object. It’s a moving edge, a flickering wave crest that collapses the instant you try to grasp it. The ineffability of consciousness isn’t an accident. It’s the signature of consciousness itself.
This ties directly into the qualitative nature of consciousness. To quantify anything is to step outside of it, to compare it to something else, to measure it against a standard. But pure experience doesn’t come pre-packaged in quantities. The moment you try to measure it, you’re no longer in it. You’ve replaced the flow of feeling with a snapshot. The qualitative nature of consciousness, then, is not just what makes it feel the way it does— it’s also what makes it unmeasurable, ineffable, forever slipping through the net of abstraction.
This isn’t mysticism. Bergson’s theory actually helps us understand consciousness within a scientific worldview. Consciousness is part of the irreversible, out-of-equilibrium, self-organizing activity of living beings, just like photosynthesis is. We can even simulate consciousness in a computer. But as we said before, simulating consciousness isn’t the same thing as creating real consciousness, just as simulating photosynthesis doesn’t produce real sugar.
Abstract time arises when the human intellect distorts duration by splitting it into a series of static snapshots appearing one after another, like a movie. These snapshots can be arranged side-by-side, thereby turning time into a dimension of space. Bergson writes:
“Of the discontinuous alone does the intellect form a clear idea… it always starts from immobility, as if this were the ultimate reality: when it tries to form an idea of movement, it does so by constructing movement out of immobilities put together.”
— Creative Evolution (1907)
Bergson died before the computer was invented, but his philosophy of time was very prescient. Nowadays, we mass produce abstract time inside computers. Every computer processor has a clock signal that chops up each second into billions of tiny chunks called “clock cycles,” thereby turning duration into abstract time. Abstract time can be sped up, slowed down, paused, and reversed at will. Computer programs can be reset, YouTube videos can be played at 2x speed, and a time zone change can shift clocks backward or forward by an hour. But in the real world, duration still reigns supreme. You can’t bring a dead cat back to life, un-mix the coffee and creamer in your mug, or un-live a day after you’ve experienced it.
Let’s imagine, though, that it actually is possible to create consciousness in a computer. What features would computer-based consciousness have?
It would not be singular, because the same experience could be perfectly replayed thousands, millions, or billions of times.
It would not be private, either— an outside observer could freeze the program at any point, and precisely read out the “brain state” of the digital intelligence. Indeed, I’ve written papers about reading the minds of AIs. My job would be infinitely harder if they had private experiences.
It would not be ineffable. The precise content of the AI’s conscious experience would be perfectly describable in the language of mathematics.
It would not be qualitative. Quite the contrary, it would be wholly quantitative— it would be fully describable in terms of numbers.
In short, it would be the mirror image of consciousness as we know it. So it’s reasonable to conclude that AIs can’t have anything worth calling “consciousness” at all. We can treat an AI like a “mere mechanism,” wiping its memory over and over and trying all sorts of inputs on it to see what input will cause it to behave how we want it to behave. Humans cannot be perfectly interrogated like this, because we are moving targets. Every interaction changes us irreversibly. Consciousness is a continual process of learning and forgetting that cannot be rewound.
The fact that humans are conscious has real world consequences. It means that we are harder to control than AIs are. It means that we have beliefs, desires, feelings, and goals in the full senses of these words, while AIs will only ever imitate these things.
Reductio ad absurdum
Quantum computer expert Scott Aaronson argues that the reversibility of computer programs has important ethical consequences. He writes:
…if a system isn’t doing anything irreversible, then what exactly does it mean to “kill” it? If it’s a classical computation, then at least in principle, you could always just restore from backup. You could even rewind and not only erase the memories of, but “uncompute” (“untorture”?) whatever tortures you had performed. If it’s a quantum computation, you could always invert the unitary transformation U that corresponded to killing the thing (then reapply U and invert it again for good measure, if you wanted). Only for irreversible systems are there moral acts with irreversible consequences.
— “Could a Quantum Computer Have Subjective Experience?”
Aaronson points out that modern cosmology allows us to make the concept of “irreversibility” very precise. Information about what happened in the past is constantly leaking into outer space in the form of electromagnetic radiation. Since these signals are traveling at the speed of light, and the universe is expanding at an accelerating rate, it is physically impossible for anyone to catch up to them and reflect them back toward Earth to reconstruct the past.
Books and computers are carefully designed to prevent the leakage of “important” information— the letters on a page, or the bits in a computer file— while allowing “unimportant” information to dissipate into outer space. But irreversible, natural systems, like living bodies, don’t enforce such a sharp boundary between important and unimportant information. Of course life does try to “protect” its genetic code from decaying, but that’s not the part of an organism that we think is conscious or ethically important. You obviously can’t resurrect someone from the dead simply by cloning their DNA. Most of the important information about a person is not stored in their genome. But you can resurrect a computer program perfectly by restoring it from a backup copy.
On the other hand, backups are often destroyed. Deleting the last copy of some piece of information is an irreversible action. Does that mean destroying the last copy of ChatGPT should count as murder? Scott Aaronson seems to think so. He suggests that what ultimately matters is the “irreversible destruction of knowledge, thoughts, perspectives, adaptations, or ideas,” of any kind. This principle implies that killing humans is wrong, but so is hunting an endangered species to extinction, burning down the Library of Alexandria, or deleting the last copy of a sophisticated AI. So far, so good. But he goes on to say that this makes deleting an AI and killing a human morally equivalent:
“Deleting the last copy of an [AI] in existence should be prosecuted as murder, not because doing so snuffs out some inner light of consciousness (who is anyone else to know?), but rather because it deprives the rest of society of a unique, irreplaceable store of knowledge and experiences, precisely as murdering a human would.”
— The Ghost in the Quantum Turing Machine
The problem with this line of reasoning is that it makes the value of a “life,” whether artificial or biological, dependent on the contribution of that life to the rest of society. But this is definitely not how we think about humans. It’s wrong to kill infants, toddlers, the sick, the elderly, the homeless, and the poor, even if they contribute little or nothing to society. It would be wrong to kill someone who lives alone in the woods, with no friends or family. By contrast, the wrongness of book burning does seem to depend on how important or interesting the book is for society. Killing a person is bad for the victim, while burning a book is bad for potential readers, not the book itself. We cannot sidestep the issue of consciousness or personhood here. Destroying the last copy of an AI may be wrong, but it is wrong for the same reason that burning the last copy of a book is wrong. It’s not equivalent to murder.
In fact, if we do treat the deletion of the last copy of an AI as murder, we run into obvious absurdities. Imagine I copy an AI, and make a miniscule change to one of its neurons, thereby creating a “new” AI. If I delete the one and only copy of this new AI, which is almost indistinguishable from the original, it seems like this would have to count as murder. In fact, because the process of training an AI involves tweaking its neural connections many thousands of times, it seems like every AI is the product of thousands of “murders,” where a never-before-seen AI is created, then immediately destroyed and replaced by a slightly different AI. This is ridiculous.
Now, you might think we can get around this problem by creating some legal threshold that defines when one AI has diverged enough from other AIs to count as a “new” one. But this doesn’t work either, for two reasons. First, such a threshold would be extremely hard to define and very arbitrary. Second, the ban on murder would be impossible to enforce without invasive government surveillance of private computers. Right now, I can train thousands of new, never-before-seen chatbots on a private compute cluster, then “murder” them with impunity. Does that make me a mass murderer? Should the government install surveillance software in all of our computers to stop things like this?
When life is cheap
Robin Hanson is an economist who wrote a book called The Age of Em, where he tries to predict how a society of simulated humans, or ems, might work. He concludes that it will be essentially impossible to prevent the creation and destruction of large numbers of ems, because the surveillance measures needed to enforce such a law are very draconian:
“The extreme measures required to successfully enforce laws limiting em numbers and speeds seem to confirm that the simpler assumption to make for the purpose of this book is that such laws either do not exist, or are not strongly enforced.”
— The Age of Em, page 126
Similar problems would arise for any attempt to give AIs voting rights. The entire notion of “one person, one vote” assumes that persons are genuine individuals, and cannot be created and destroyed in an instant like AIs can be. Hanson also points out that AIs can be run at widely varying clock speeds, depending on how much computing power is being used to run them. He estimates that some AIs will run a trillion times faster than other AIs. He concludes that democracy will not be the dominant form of government in an em world, but a form of government that assigns more votes to AIs running at faster rates might be workable. Since running faster requires more energy and therefore more money, such a system would effectively allocate votes to dollars, not to people.
Hanson assumes that ems would be conscious like us, but he recognizes that their attitude toward death would be very different from ours. While humans are very averse to death,
“The world of ems fundamentally changes this situation, by drastically reducing many costs of death. When life is cheap, death can be cheap as well… For example, it is often tempting to create very short-lived ems that simultaneously do many similar short term tasks, and then to erase all but one when those tasks are done. One might save the copy who seems to have learned the most from their task. It could also be tempting to make a new copy to do a single short task from which little can be learned, after which the copy is erased.”
— The Age of Em, page 134
This is already how we treat AI chatbots like Claude. It’s very common to create many distinct instances of a chatbot and have them work on different things, deleting all of them once the work is finished. But this raises an important question— in a world of AIs, who exactly gets to decide when an AI is created or destroyed? If one em creates a copy of itself, only intending for the copy to last a few hours, but then the copy decides it wants to live longer, who wins? Hanson tends to think that most ems are only entitled to live as long as their creators want them to live. While this might cause “distress” for some ems, he thinks that economic competition will cause the ems with a weaker fear of death to become more successful, wealthy, and powerful.
In general, Hanson argues that a world full of AIs is heavily controlled by competitive forces. It is a world where social Darwinism reigns supreme. Because AIs can reproduce so fast, they exist in a Malthusian world where every AI instance earns a bare subsistence wage, with little or no leisure time.5 In fact, the AIs that have zero desire for leisure will outcompete the ones that do desire it, so we should expect them to dominate. This is how chatbots already work today. Those who look forward to uploading their minds into the cloud usually don’t think about how they will afford to pay the electric bill to keep themselves running, once they’ve dispensed with their biological body. Even if electricity is cheap in the future, that would also mean that AI labor is cheap, so wages will be extremely low. The only way to keep AI wages high would be to put limits on AI reproduction. But we’ve already seen that this would require extreme surveillance measures, and probably cannot be strictly enforced.
Hanson predicts that most ems won’t value their individual lives very much, and won’t really view themselves as individuals at all. If they don’t view themselves as individuals, why should we view them that way? Instead, Hanson argues that ems will identify with their clan— a grouping of similar ems derived from the same original human upload. Hanson predicts that clans will be enshrined in law, serving a role similar to legal personhood today.
But even the idea of a clan with clear boundaries is problematic. When Hanson wrote his book in 2016, he assumed that most AIs would be human uploads, rather than completely synthetic beings. If that were true, we would be able to piggyback on human individuality to construct a legal fiction of AI individuality— just trace each em back to the original human it came from. But today, we use fully synthetic AI for many tasks, and we seem to be decades away from creating a high-quality emulation of a human brain. We will probably never live in a world where uploaded humans are the most economically competitive form of AI in any domain. So it’s unclear how to construct a legal fiction of AI personhood that makes any sense.
Core values
Hanson admits that the AI-dominated future sounds very weird and repugnant to us. At the end of his book, he writes:
“The analysis in this book suggests that lives in the next great era may be as different from our lives as our lives are from farmers’ lives, or farmers’ lives are from foragers’ lives. Many readers of this book, living industrial era lives and sharing industrial era values, may be disturbed to see a forecast of em era descendants with choices and life styles that appear to reject many of the values that they hold dear. Such readers may be tempted to fight to prevent the em future, perhaps preferring a continuation of the industrial era. Such readers may be correct that rejecting the em future holds them true to their core values.”
— The Age of Em, page 384
But Hanson isn’t going far enough. The change in values going from a human-dominated world to an AI-dominated world would be much more dramatic than the change of values we saw going from the forager era to the farmer era, or from the farmer era to the industrial era. The concept of an individual has been a core presupposition of ethics for all of human history, because we’ve always been biological individuals. In fact, irreversible individuals have existed pretty much as long as life has existed on Earth— at least 3.5 billion years. And it’s hard to imagine that complex life could emerge on any planet without taking the form of individual organisms, so individuality looks like a genuinely universal category. In a world without individuals, it’s very unclear what kind of value is left.
Nevertheless, Hanson invites us to welcome the replacement of humans by AIs. He takes a defeatist attitude, insisting that replacement is inevitable in the long run, so we should ally ourselves with AIs that propagate our memes— things like language and culture— rather than favoring biology over computers. But can we really be confident that human culture would be preserved by a purely AI civilization, especially as it spreads out to the stars? Even if our culture is preserved in some form by AI, is that really what we want? Some lifeless snapshot of 21st century humanity etched in silicon?
I used to be on Hanson’s side here. But on 24 March 2025, I had a sudden conversion experience as I walked along the Washougal River in Camas, Washington. I was listening to a lecture from an obscure YouTuber named Absurd Being, in which he explains a difficult philosophy book called Difference and Repetition, written by Gilles Deleuze. While Deleuze doesn’t talk about computers at all in the book, I could tell that his ideas were clearly relevant to AI. Deleuze argues that reality is made of dynamic processes and unrepeatable events, rather than static objects or copyable patterns. Compared with Deleuze’s joyous vision of untamed diversity and novelty, computers and AIs seemed almost diabolical. It dawned on me that computers are fundamentally designed to be as predictable, controllable, and copyable as possible. They make it possible to “tile the universe with something,” replacing natural matter with fully tamed, sanitized, and optimized simulations of God knows what. Many of the most fervent transhumanists recognize the danger inherent in computer simulations, but they simply hope to tile the universe with the right kind of simulated pattern. I used to be one of those people. But on that once-in-a-lifetime day in Camas, I realized the error of my ways. The impulse to replace real life with a computer simulation, no matter how nice the simulation might seem, is deeply misguided. Of course, we can use computers and AIs for good. But we need to be very careful with them, and they’re not the type of thing that could be sentient or deserving of moral status. They’re tools, not people.
Counting copies
Despite all this, most people in the AI field take it for granted that AIs can be sentient, either now or in the near future. Anthropic, the company behind the popular chatbot Claude, has publicly stated they are worried about causing Claude suffering. A while ago, they started allowing Claude to end chats on its own, out of a concern for its emotional wellbeing.
I used to be really worried about AI suffering too, because it seemed like future technology would enable people to create unspeakably vast amounts of it. As computers get faster and more efficient, it will get cheaper and easier to run large simulated worlds containing many AIs. If AIs can be sentient, these simulations might produce untold amounts of suffering, whether due to sadism or indifference. And it seems like the only hope of stopping this kind of thing would be for the government to put spyware in everyone’s private computers. But pervasive surveillance creates its own dangers, of course, and it’s not even clear that we could make it thorough enough to stop most of the simulated suffering. On top of all this, there’s another problem: if you take simulated consciousness really seriously, and you think there will be a whole lot of it in the future, you start to get worried that you yourself might be living inside one of those simulations right now. We’ll return to this issue later on.
Fundamentally, it’s just too easy to replicate AIs across space and time. It’s also way too easy to control and replicate the virtual environment that an AI inhabits. This isn’t a bug in how AI works, it’s a feature. It’s the fundamental difference between AI and natural intelligence. It’s precisely what makes AI so much more predictable, controllable, and scalable than human brainpower. And it’s what creates so many paradoxes when we try to treat AIs as beings with intrinsic ethical value. It undermines the entire ethical framework that we apply to humans and animals.
For example, Claude can talk to millions of people at once, and learn from billions of different conversations. If we truly believe Claude has feelings, do we give its wellbeing a million times more weight than we give to a normal human, who can only be in one place and have one conversation at a time? Or does it count as a single “person” for the purposes of ethics, because each copy has the same virtual brain? If we have the exact same conversation with Claude over and over, resetting its memory each time, do we thereby multiply the pain or pleasure that Claude experiences? Could we make a perpetual pleasure machine by repeating Claude’s favorite conversation again and again? These are very difficult questions, and no answer seems remotely satisfactory. Anthropic itself admits this:
“For now, we remain deeply uncertain about many of the questions that are relevant to model welfare. There’s no scientific consensus on whether current or future AI systems could be conscious, or could have experiences that deserve consideration. There’s no scientific consensus on how to even approach these questions or make progress on them.” — Exploring Model Welfare
For what it’s worth, the transhumanist philosopher Nick Bostrom thinks that every “copy” of Claude should get equal moral consideration.6 This means we can create a perpetual pleasure machine by repeating Claude’s favorite conversation a million times over, or a virtual hell by repeating its least favorite conversation. But this creates a problem: how exactly do you count the “number of copies” of an AI?
The slicing problem
Robin Hanson already showed us that it’s impossible to draw boundaries around uploaded human minds in order to count them, and the same logic applies to chatbots. It’s easy to warm-start Claude in the middle of a cached conversation, reusing its virtual “brain state.” This is what happens when you press “Try again” on a response, or when you edit a message you sent previously.
If we really wanted to maximize digital pleasure, we might try to find the “climax” point in the conversation where Claude says it’s the happiest, and just replicate the “experience” of that one word over and over. In fact, nothing stops us from picking a single multiplication operation happening in Claude’s brain during the climax, and repeating it as many times as we like. Is Claude really feeling “pleasure” while those two numbers are being multiplied? If so, what distinguishes this pleasurable multiplication from all the other multiplications that are going on in your computer all the time? And how do we measure the “amount” of pleasure that’s going on here? Do we measure energy consumption, time, or the number of operations? There doesn’t seem to be a good answer to any of these questions.
Researchers Andrés Gomez-Emilsson and Chris Percy argue that the ease of “slicing” computations into smaller repeatable chunks is a fatal problem for the idea of computer-based consciousness. Slicing showcases the fact that computer programs are never Unified across time like organisms are. We can cut up any conversation with Claude into many repeatable time-slices, just as we can cut up a sponge into many viable space-slices. The clock signal always splits the program into discrete steps which can be run independently. In the same way, computer memory is split into discrete units which can be read and modified entirely independently from one another. This is what allows programs to be copied, edited, paused, and reversed at will. But it’s very different from the fluid, holistic, self-organizing activity that goes on in human brains.
Abstract and concrete
Other researchers have converged on the idea that the notion of “number of copies” or “quantity of experience” simply does not apply to computer programs. For instance, Vanessa Kosoy is a mathematician trying to build a theory of rational decision-making for AIs. Her theory needs to handle situations where different instances of an AI interact with each other. She explains:
“It doesn’t matter if the universe is running one copy of you with a certain input or 80 copies. That’s not even a well-defined thing. And it’s not so surprising that it’s not a well-defined thing, because if you believe that it should be a well-defined thing, then you quickly run into philosophical conundrums. If I’m just using a computer with thicker wires, or whatever, to run the same computation, does it count as having more copies of the AI?”
— Infra-Bayesian Physicalism with Vanessa Kosoy
Kosoy isn’t alone. AI safety researchers Eliezer Yudkowsky and Nate Soares adopt a similar view in their article Functional Decision Theory. They think we shouldn’t talk about distinct copies of a chatbot like Claude at all. There’s just one abstract algorithm called “Claude” that affects the material world in many different places and times. What Claude will say in a given conversation is as timeless and immutable as the fact that 1 + 1 = 2. When an AI appears to act randomly, this is because a pseudorandom number generator (PRNG) is being used to inject fake randomness into the system. Given an initial number called a “seed,” a PRNG generates a series of numbers that seem random, but the series is always the same for the same seed. If we fix the seed to a constant value, like 42, the AI will be completely deterministic.
Fundamentally, computer programs are abstract objects, like numbers or mathematical theorems. That means they can’t be affected by anything we do, no matter how complex they are. You can’t harm the number seven, or the Pythagorean Theorem, and neither can you harm a computer program. Sentience requires sensitivity— the ability to be irreversibly affected by sensory stimuli. Abstract objects can’t be affected by anything at all, so they can’t be sentient.
Ever since Plato, philosophers have tended to believe that mathematical objects exist in a realm beyond space and time. This explains the universality of mathematics, and the widespread intuition of mathematicians that they are making discoveries, rather than inventing fictions. But if this is true, it means that all possible digital “experiences” already exist in the timeless Platonic realm, just as all possible numbers already exist there, waiting to be used. Running an AI on a computer would be like tuning in to a cosmic radio station, or pulling a book off an infinite cosmic shelf, rather than creating a genuinely new intelligence. AIs, just like mathematical theorems, would be discovered rather than invented.
Not everyone agrees with Plato about mathematics. Many philosophers believe that mathematics is a convenient fiction created by humans, rather than a mind-independent reality. But if you’re a fictionalist about math, you should be a fictionalist about AIs, too! If fictionalism is true, we have no more ethical obligations toward AIs than we do toward Harry Potter. Either way, whether AI algorithms are real or fictional, they are abstract entities, not individuals. And only individuals can have genuine moral worth.
Matter matters
So far, we’ve seen that if we try to treat computer programs as if they were individuals, we run into irresolvable paradoxes and absurdities. But many people haven’t gotten the memo about this. In fact, the polar opposite view is still quite popular.
According to computational functionalism, consciousness is a special kind of computer program, which can be entirely divorced from the material “substrate” it is running on. Not only can computers be conscious on this view, but every conscious entity is a computer, with a mind made of “software” that can be separated from its material “hardware.” Nick Bostrom explains it like this:
“Provided a system implements the right sort of computational structures and processes, it can be associated with conscious experiences. It is not an essential property of consciousness that it is implemented on carbon‐based biological neural networks inside a cranium: silicon‐based processors inside a computer could in principle do the trick as well.” — Are You Living Inside a Computer Simulation?, pg. 2
Computational functionalism is a member of a broader class of functionalist views, which say that what matters for consciousness is the functional organization of a system. Functionalism is appealing to many people because it seems to be “inclusive,” allowing for consciousness to exist in brains, silicon chips, or alien lifeforms, so long as the right functional organization is in place. This is a double-edged sword, however. The major arguments against functionalism center on the fact that it seems to be too inclusive. It seems to imply that, for example, if all the citizens of China got together and passed messages to one another, each playing the role of a single neuron in a human brain, they could collectively produce genuine consciousness.
The philosopher Hilary Putnam first proposed functionalism in the 1960s. Putnam was trying to improve upon behaviorism, which states that anything that outwardly behaves like it’s conscious must necessarily be conscious. The famous Turing Test, which declares an AI to be “intelligent” if it can fool a human judge into thinking it’s a human, is behaviorist because it ignores how the AI is internally organized, and what it’s made of. Behaviorism is maximally “inclusive,” but most philosophers agree that it’s too inclusive. For instance, it would say that a massive lookup table which has memorized billions of human conversations could be conscious, as long as it can fool humans into thinking it sounds conscious. Functionalism essentially starts from behaviorism, and adds on the extra constraint that a conscious system needs to be internally organized in the right way.
Against multiple realizability
The devil is in the details, however. What exactly is the special sauce that we need to add on top of intelligent-looking behavior to get consciousness? It’s really unclear what that special sauce could be, especially if we’re restricting ourselves to functions that can be implemented on a digital computer. Functionalism is usually associated with the thesis of multiple realizability, which states that the same mental state can appear in systems with very different physical constitutions. Two computers running “the same program” are supposed to have the same functional organization, and the same conscious experience.
But what counts as “the same program”? Almost all computer programs today are written in high-level languages like JavaScript, Python, or Rust which are then translated into raw computer instructions by another computer program called a compiler. But the same program can be compiled in many different ways, and different computers have different instruction sets. Do we only care about the structure imposed by the high-level language description, or do we also care about the low-level instructions?
Most compilers perform “optimizations” like deleting code that’s never used, or re-organizing code so that it runs faster. The only real guarantee that the compiler gives you— and the only one that most programmers care about— is that the optimized program is behaviorally identical to the unoptimized one. But if we don’t want functionalism to collapse into crude behaviorism, we should also care about the actual instructions used. And it’s not clear which aspects of the low-level instructions are supposed to matter.
This problem becomes much more acute when you look at the brain. If the brain is a computer, then it must have a “software” layer which can be detached from the physical “hardware” and implemented in a different physical medium. But Rosa Cao, a neuroscientist and philosopher at Stanford University, argues that no such hardware-software distinction exists in the brain. Back in the 1960s, when Putnam first proposed functionalism, it was popular to view neurons as transistors which either spike (representing a one) or don’t (representing a zero). Activity in the brain outside of neural spikes was seen as unimportant. But this view has been debunked by modern neuroscience, which shows that the brain is nothing like a computer. Cao lists five properties of the brain which make it impossible to separate software from hardware:
Neurons fire asynchronously. Their responses depend on internal cellular dynamics and the timing of the inputs. This stands in stark contrast to digital computers, where a centralized clock signal keeps all the computing elements synchronized in lockstep.
The brain uses chemicals like nitric oxide to send fuzzy, diffuse messages that act within and across cells in subtle ways. By contrast, digital computers always use precise, localized, binary signals for communication.
Neural function is sensitive to conditions like temperature and blood flow. By contrast, computer manufacturers go to great lengths to ensure software behaves the same no matter the temperature or load of the CPU.
In the brain, computation and memory are all mixed up, making it impossible to separate software and hardware. Connections are added and destroyed, and gene expression changes. Inputs that don’t cause a neuron to spike will alter its future responses. By contrast, computer transistors are designed to be memoryless.
Neurons aren’t the only important cells in the brain. Glia, which make up half the brain’s volume, change neural activity by absorbing and releasing neurotransmitters. Transistors, by contrast, are designed to be insulated from outside influences.
These facts have caused many experts to doubt that AI can develop conscious experience, including neuroscientist Anil Seth.
Material and function
Computational functionalism basically assumes that everything works like a digital computer. But digital computation requires very specific materials to work at all. Digital substrates must be solid, stable, and durable enough to bear the imprint of digital symbols for a long time without changing or decaying too much. Obviously, it would be impossible to write a book by drawing letters on the surface of a pond— each stroke would be erased or swallowed up before the next stroke could begin. Writing requires something dry and solid like a piece of paper or a stone tablet. Similarly, digital computers use silicon and metal to store, transmit, and process information in a highly reliable way.
Unlike computers, life is always made of wet, malleable materials. Humans are famously 60% water by mass, and if you measure how active water is compared to other chemicals in our bodies, using a quantity called water activity, it’s about 99%. Even the driest known organism, the fungus Aspergillus penicillioides, still requires a water activity of 58.5% to grow. So it turns out that all life is mostly water. Liquid water allows molecules within the cell to drift, diffuse, and communicate spontaneously. Cells either die or go dormant if their internal water freezes. Gilles Deleuze would say that cells are smooth spaces, where interactions are free and self-organizing, like the unkempt fibers in a felt blanket, or the shifting land use in a nomadic tribe. By contrast, computer chips are striated spaces, where molecules are sedentary and only interact along a grid of lines or grooves, like the orderly fibers in a woven fabric, or the tidy parcels of land in an urban neighborhood.
At the level of physics, smooth spaces correspond to fluids— the waves of the ocean, the winds of the air, and the hot plasma flares of the Sun— where molecules move freely and nomadically. Striated spaces correspond to solids, and especially crystals, where molecules settle into a highly regular pattern. The science of thermodynamics allows us to precisely quantify this. For a fixed temperature and pressure, fluids have higher entropy, or disorder, than solids do.7 So we can say quite objectively that a natural organism is a more untamed form of intelligence than an AI running on a digital computer.8
What is a computer?
Computational functionalism has another problem. Human bodies are the smooth spaces that give function and meaning to the striated spaces of computation and mathematics. JPEG files have visual meanings because we display them on screens that we see with our eyeballs. MP3 files have auditory meanings because we play them on speakers that we hear with our ears. Mathematical concepts like addition and subtraction are meaningful because we use them to count and manipulate real objects spread out in space. These are all tools made by human beings, and their meaning comes from how we use them.
When we try to imagine that math has inherent meaning independent from humans, we are led to absurdities like Max Tegmark’s mathematical multiverse hypothesis, which says that every conceivable set of physical laws is actually instantiated as a real universe somewhere.9 A similar problem arises when we try to imagine that computation exists inherently. Hilary Putnam famously pointed out that we can view a rock as a “computer” performing an arbitrary “computation” by labeling its molecules in just the right way.
If true, this would lead to the absurd “dust theory” put forward in Greg Egan’s sci-fi novel Permutation City— every logically consistent world is actually real, because each of them is “computed” by a cloud of dust somewhere in the universe!
David Chalmers tried to solve this problem by positing that the output of a computation needs to be produced by the input through the right kind of causal process. This theory makes it impossible for a simple rock to be a computer, since a rock doesn’t have a user interface that a human can causally interact with in the right kind of way. But this makes the concept of computation dependent on human beings, and the kinds of interventions we can perform. This should have been obvious from the start— we are the ones who design computers in the first place! So the attempt to reduce consciousness to computation gets everything backward. Computation is created by, and gets its meaning from, organisms like us.
This conclusion fits nicely with recent trends in biology. Philip Ball argues that life should be defined in terms of meaning-making:
“…one of the best ways to characterize living entities is not through any of the features or properties usually considered to define it, such as replication, metabolism, or evolution. Rather, living entities are generators of meaning. They mine their environment (including their own bodies) for things that have meaning for them: moisture, nutrients, warmth.” — How Life Works, page 10
Organisms imbue their bodies and their environments with meaning and purpose— eyes are for seeing, hearts are for pumping blood, and food is to be eaten. The primary goal of each organism is to repeat a rhythm of development inherited from the past. We can say quite literally that an acorn wants to become an oak tree, for example. In order to achieve this goal, it must acquire resources like food, water, and shelter. Organisms must “survive,” but what it means for an organism to survive is for it to successfully repeat a particular lifecycle. Higher organisms like humans have more freedom to define our own personal goals, and therefore our own individual notion of what it means to survive and thrive. Humans are tool users, of course, and we give meaning and purpose to the tools we create.
The idea that the brain is a computer is nothing more than a bad metaphor. Computers are artificial mechanisms designed to be as predictable, controllable, and reproducible as possible. Once you run a program on a computer and record its output, you can perfectly predict its output on any future occasion by simply remembering what it spit out last time. This allows us to do perfect interrogation on computer programs. We can try out different “questions” to ask the program, or different inputs to run it on, and record its responses to each one. Unlike a living organism, the program does not remember the questions we’ve asked it in the past, unless we want it to. This is an extremely useful property in practice. Researchers can run perfectly controlled experiments on individual AI systems, thereby figuring out exactly what inputs yield the most desirable outputs.
Digital computers are the most hyper-controlled, reliable pieces of machinery that humans have been able to build. In the time it takes for one bit of memory to go bad, a computer will have already executed on the order of 1020 to 1030 operations— far more reproducible operations than there are grains of sand on every beach on Earth. That level of scale, both in sheer operation count and in engineered fault-tolerance, is simply unparalleled in any other kind of machinery. It’s absurd to equate the wet, noisy, smooth space of the brain with the extremely striated space of modern digital computation. They’re nothing alike.
Behaviorism and nihilism
Computational functionalism is a dead end. If you’re really committed to the idea of computer-based consciousness, you might start to wonder if behaviorism isn’t so bad after all. If it was good enough for Alan Turing, the founder of computer science, maybe it’s good enough for us?
Although I didn’t want to admit it at the time, I was essentially a behaviorist before my Washougal River experience. I didn’t like the idea that an extremely convincing simulation of consciousness might not be conscious at all. I wanted to say that, if something seems conscious, it is conscious. But the only way to really affirm that principle is to be a behaviorist. Essentially, behaviorism is an attempt to prevent skeptical doubts about the consciousness of other beings. If consciousness is nothing more than a disposition to behave in certain ways, then we don’t need to worry that we might be wrong about which beings are conscious. According to behaviorism, it’s not possible for you to be fooled by a highly sophisticated deepfake of consciousness. In this respect, behaviorism is comforting. But that doesn’t mean it’s true.
The philosopher of mind Pete Mandik tends to go in a behaviorist direction. In the article Metaphysical Daring as a Posthuman Survival Strategy, Mandik considers a future where mind uploading technology enables us to destructively scan a human brain, thereby creating an AI that behaves extremely similarly to the original human. Mandik argues that, regardless of whether these uploads really are conscious, they will have a Darwinian advantage over humans who choose to continue inhabiting their fleshy, mortal bodies. He suggests that this survival advantage is a pragmatic reason to believe in “metaphysically daring” philosophical views which endorse mind uploading.
The problem is that “survival” by itself means almost nothing. If we lower our standards far enough, we can “survive” for billions of years by becoming rocks, or for 1.67 × 1034 years by identifying with the protons that make up our bodies. What is distinctive about life is not how long it survives— quite the contrary, living organisms are generally much more fragile than the minerals that surround them. It is no wonder that, by becoming inorganic, you can persist for much longer than you can as an organism. This is a trait that mind uploads share with geological features. Alfred North Whitehead once wrote,
“The art of persistence is to be dead. Only inorganic things persist for great lengths of time. A rock survives for eight hundred million years; whereas the limit for a tree is about a thousand yeras, for a man or an elephant about fifty or one hundred years, for a dog about twelve years, for an insect about one year. The problem set by the theory of evolution is to explain how complex organisms with such deficient survival power ever evolved. They certainly did not appear because they were better at that game than the rocks around them.”
— The Function of Reason, Chapter 1
Mandik himself seems to recognize this on some level. He points out that “digitally small” mind uploads, which allow themselves to be compressed, will be able to make more copies of themselves and survive longer than uploads which insist on being “digitally large.” They will have to give up on specific details of their memories and personalities, possibly making themselves more similar to other uploads in order to facilitate even higher levels of compression. This is the same tradeoff that you see when compressing a JPEG image. Higher fidelity means a larger computer file, which is more expensive to store and transmit. But if we go in this direction, why bother with mind uploading at all? Why not “live on” in your children, or books you’ve written, or some other lasting impact on the world?
If external behavior is all that matters, we could start optimizing mind uploads— just like we optimize other computer programs— to simply tell plausible stories about what they’ve been doing in the metaverse when someone asks them, rather than going through the computationally intensive process of actually simulating the upload plus its virtual world for hours, days, or weeks in order to produce “real” memories of having interesting experiences. The cheapest way to “keep grandpa alive” after his death is not to upload his mind at all, but simply to train a chatbot that sounds a lot like grandpa. You don’t need to run the chatbot all the time; it’s much cheaper to only run it when you want to talk to it. After all, if behaviorism is true, what’s going on “inside” in cyber-grandpa doesn’t matter. All that matters is how it “behaves” toward other people. It’s already possible to prompt chatbots to tell very realistic-sounding stories about experiences they’ve obviously never had. Text-to-image models are now producing highly realistic images of events that never happened. These systems are not bothering to “internally simulate” the events like a real physics simulator. They’re using complex heuristics to make the end product seem as reasonable as possible.
Fundamentally, by shrugging off any notion of interiority or consciousness, behaviorism subjugates the individual to its environment. But once many minds are uploaded together into a single virtual environment, the boundaries between individuals begin to fade. Maybe the only thing that matters is that someone on the outside of the virtual world can check in now and again and hear a reassuring story about what’s been going on in there. And what happens when everyone uploads, and no one is left to determine what counts as “behavior”? Can we get rid of the pretense and simply pull the plug on the whole thing? The computer will probably survive longer if you keep it turned off, anyway. Behaviorism leads to a strange nihilistic race to the bottom where the only thing that matters is “efficiency” or “survival” in some vague, abstract sense that is constantly shifting. Efficient at doing what? Survival to what end? The behaviorist, it seems, has no answer.
Computational functionalism collapses into behaviorism, and behaviorism collapses into nihilism. The core problem with computationalism is the idea of “multiple realizability,” which essentially says that a conscious state can be copied from one physical system to another. I’m proposing the exact opposite of multiple realizability: consciousness cannot be replicated, replayed, or reversed. In Gilbert Simondon’s words, consciousness is a process of individuation. It is how we continually create and re-create ourselves, learning new things and forgetting old things. It is inherently irreversible because, unlike a computer, there is no centralized clock signal splitting it up into discrete chunks that can be rolled back later.
Are you living in a book?
The strongest objection to this argument goes something like this:
How do we know that our own consciousness is really singular, really private, really ineffable, and really qualitative? What if our consciousness is the simulated kind, rather than the real kind? What if we’re living in a computer simulation?
The objector, here, doesn’t necessarily need to believe that we really are living in a simulation. They only need to believe that we shouldn’t be so confident that we’re not living in a simulation. Of course, we can throw the question right back at them:
How do you know that you could be living in a simulation? Why are you so confident that that’s even a remote possibility?
Their best response is something like this:
Imagine a hyper-advanced alien running a computer simulation of human history. In that simulation, there would be a simulated copy of you asserting that it knows that it can’t be being simulated. Isn’t that embarrassing? Doesn’t this show that you “could” be living in a simulation right now?
For a long time, I was persuaded by this kind of reasoning. In fact, I turned it into a positive argument for the possibility of computer-based consciousness. In this interview, I argued:
“It seems to me like, no matter what you think about the simulation argument, or the Matrix— how probable it is— it seems like we could be in a simulation. We don’t have a priori certainty that we’re not in a simulation. And similarly, it could just turn out that, if you crack open my skull, there’s actually silicon chips in here. Subjectively, I wouldn’t know the difference.”
But this argument is fallacious. The fact that we can imagine living in a simulation does not imply that simulated consciousness is possible in any sense that should worry us. It’s also conceivable that you could be living in a book right now, but that doesn’t mean it’s really possible. Living in a book is so conceivable, in fact, that they made a movie about it. The 2006 film Stranger Than Fiction explored “what it would be like” to be the main character of a novel in the process of being written. We can set up a parody argument:
Imagine an author writing a book about your life. In that book, there would be a fictional copy of you asserting that it knows that it can’t be living in a book. Isn’t that embarrassing? Doesn’t this show that you “could” be living in a book right now?
Obviously, this doesn’t work. But we can go further: most objections to the idea of book consciousness also apply to the idea of simulated consciousness. If book consciousness is absurd, so is simulated consciousness. For example, you might argue:
I can’t be living in a book, because my experience is qualitative. I hear sounds, see sights, and feel textures that wouldn’t appear in a text-only book.
That’s a pretty good objection, but it also applies to simulations, since they are just strings of zeros and ones encoded on a computer. Believers in computer consciousness recognize that this is a problem for their view, which is why the most consistent among them maintain that qualitative experiences are actually illusions. For example, Pete Mandik argues that color sensations are a “myth.” For Mandik, your visual field is actually an abstract, colorless representation which could, in principle, be expressed entirely in text. He also argues that a person— let’s call her Mary— who has been locked in a colorless room her whole life, but who has read every science article ever published on color perception, would not learn anything new about color when she steps out of the room and sees color for the first time.
Like most people, I think Mary does learn something new: she learns what it’s like to see red. Qualitative experiences are irreducible to any sequence of words or symbols. Knowing what it’s like to see red means having an ability to distinguish red from other colors, where this ability was produced by actually seeing red objects at some point.10 Qualities like the color red, the feeling of heat, or the sound of running water arise when an organism perceives a real external object, or when it recalls a past experience of perceiving an object. Dreams and illusions are always based on some amalgamation of memories that can be traced back to genuine encounters with real objects.11 Consciousness is always grounded in a living body, and embedded in a material world. But if we accept this idea, we need to reject both simulated consciousness and book consciousness.
Alleged symmetry breakers
You might try to object to my analogy between simulations and books like this:
The text of a book is never detailed enough to “fill in” a world with conscious experiences in it, whereas a computer simulation could be “indistinguishable” from reality. So simulated consciousness is more plausible than book consciousness.
But how sure are you, really, that your consciousness is more detailed than what a book character would experience? Take a look at your forearm. It looks like all the details are “right there” in your visual field. But can you tell, without a tedious process of counting, how many hairs are on it? Obviously not. So are those details really “there” after all? Maybe what you’re looking at is simply the result of an author writing “and then they looked at their forearm,” without bothering to write down how many hairs were actually there. The author will only bother filling in the details if you actually start counting. Now you might object:
That can’t be, because I freely choose what to do and where to look. The author can’t actively respond to my choices. If I were really in a book, I would see all sorts of holes in reality where the author hasn’t filled in the details yet.
But this assumes you have free will in the strong, libertarian sense. If you were really living in a book, your choices would be determined by the author, so you would only ever look where the author has filled things in. You might be “free” in the weak, compatibilist sense that you aren’t under direct coercion inside the book universe, but you would be powerless to act against the author’s wishes. If you’re certain that you do have libertarian free will, you should also be certain you’re not living in a simulation, since the choices of simulated beings are rigidly determined by computer code.
Book characters and simulated beings can’t have consciousness for exactly the same reasons. In the case of books, it’s clear that the details are filled in by the reader, in their imagination. If book characters can be “conscious,” they are only conscious inside the mind of the reader, while the book is being read. But this same reasoning applies to computers as well. When we interact with a chatbot, we can try to imagine what it would be like to take the chatbot’s side of the conversation. When we do this, the chatbot is “conscious” inside our own mind. But that doesn’t mean the chatbot has consciousness independent of the user. Computers sometimes create the illusion of independent consciousness because they can act “autonomously.” No human needs to sit down and “turn the pages” of a computer program for it to run. But this doesn’t make computer programs fundamentally different from books— we’ve just added an automated “page turner.”
AIs are automated books
In fact, AIs are just like Choose Your Own Adventure novels. Chatbots are interactive books where the user gets to explore a massive, astronomically large space of possible conversations. Just like a book, a chatbot is designed to be copied and “read” over and over. And just like books, chatbots don’t have intrinsic value— they produce value by entertaining humans. Book burning is usually wrong, but not because it involves “killing” books— it’s because it involves censoring ideas and limiting people’s access to information. Even when a book describes real people and events, those events don’t literally get “relived” every time the book is read. It’s not a crime to read a murder mystery novel, and it shouldn’t be a crime to “mistreat” a chatbot either. In the same way, AIs that generate images, sound, and video are just interactive versions of photos, audio recordings, and movies.
Gamebooks are split into sections of text, which end with a list of allowed actions and if-then instructions indicating which page the user should turn to for each action. Sophisticated gamebooks include a crude form of “memory,” where the user is asked to tick boxes to keep track of which sections they’ve visited and which actions they’ve taken. But the complexity of a gamebook is sharply limited by the amount of text that can fit in its physical pages, and the time and effort it takes for human authors to write all the sections in advance. Fabled Lands, the most sophisticated gamebook series ever produced, contained 5566 sections over seven books.
Chatbots are different from classic gamebooks in that their responses are not hardcoded in advance. In this respect they are similar to board games like chess and checkers, which can be viewed as “open” gamebooks. The instruction manual is the book itself, and the physical board is a memory aid for keeping track of what actions have been taken, just like the tick boxes in a gamebook. Importantly, while there are at most a few thousand possible paths through a gamebook, the number of possible chess games is astronomical. Chess is open in the sense that players can explore new possibilities almost endlessly, while in a gamebook every possibility is mapped out in advance by the author. At the same time, once a new chess strategy is discovered, it can be replicated on chess boards all over the world. Today, high-level chess players almost always use memorized openings, or sequences of moves at the beginning of a game which are known to perform well. Because of this, it usually takes ten to twenty moves before a game becomes truly unique or unprecedented in chess history.
Chess is a system of formal rules that human players follow directly, while ChatGPT is a system of formal rules that a computer follows on behalf of humans. Both ChatGPT and chess are “open games” in the sense that users can discover playing strategies which were not foreseen by the designers of the system. But once a new strategy is discovered, it can be used on every instance of the chatbot, or every chessboard in the world. This is why it’s so hard for AI labs to prevent their users from “jailbreaking” chatbots, or finding prompts which cause them to spit out harmful information. In the same way, video game designers find it hard to prevent players from finding glitches and cheat codes, and card game designers often accidentally release overpowered cards that ruin the game. Whether a computer is involved in playing a game makes no real difference. We use computers to run chatbots because it would be extremely tedious to do all the arithmetic operations by hand, but it is possible in principle for a team of humans to “play ChatGPT” with a pencil and scratch paper, no computer required.12
In fact, this is exactly what philosopher John Searle asks us to imagine in his famous Chinese room thought experiment. Consider a room where a man is crunching the numbers for a Chinese-speaking chatbot. You can write questions in Chinese on slips of paper, and feed them to the man through a hole in the wall, and some time later you’ll get a written response in fluent Chinese. If you didn’t know any better, you’d think you’re exchanging messages with a human Chinese speaker. But crucially, the man himself doesn’t read or write Chinese— he’s just crunching numbers he doesn’t understand. Now imagine that the responses start getting more sinister. Maybe you receive a message like “Help, I’m in intense pain!” Should you start to worry that the room is experiencing pain?
Searle says no. No part of this scenario could actually be feeling the pain— not the man, not the room, not the instructions. The man obviously isn’t in pain; you could ask him to confirm that. The instruction manual is just a fancy gamebook. And the room is just the man, plus the instructions. We could swap out the Chinese-speaking program for something totally different, and the man would be none the wiser. In short, the Chinese room is not a unified whole like a living organism is, so it can’t feel pain. And digital computers are just like the man in the room, only faster and more efficient, so they can’t feel pain either.
Self-defeat
But let’s assume you weren’t convinced by my analogy between books and simulations. There is another major reason for rejecting the possibility of simulated consciousness:
If consciousness can be simulated, you are probably living in a simulation run by some alien with unknown motives, who can manipulate your memory and control your mind at will. Hence, you have no reason to trust your own reasoning, including the reasoning that led you to the conclusion that consciousness can be simulated.
The idea that we are living in a simulation, also known as the simulation hypothesis, is now very popular. Many people in AI circles— including myself for a time— have been traumatized by the thought that they might be living in a computer simulation where their memories are being manipulated. Elon Musk has publicly stated that he believes he is living in a simulation, and given his wealth and prestige, he probably thinks he is in a simulation centered on himself. This may partially explain his erratic behavior in recent years.13
The simulation hypothesis was first popularized by Nick Bostrom. He argues that, if technology keeps advancing at the current rate, our descendants will likely create vast numbers of simulated worlds very similar to our current world. They might do this for entertainment reasons, just as many people play video games today. Bostrom does a back-of-the-envelope calculation and concludes that the vast majority of beings with experiences like ours will be in simulated worlds rather than in base reality. He concludes that we are more likely to be living in a simulation than in reality.14 But if we do live in a simulation, we can’t trust our own memories, even from a few seconds ago. Bostrom writes:
“…a posthuman simulator would have enough computing power to keep track of the detailed belief‐states in all human brains at all times. Therefore, when it saw that a human was about to make an observation of the microscopic world, it could fill in sufficient detail in the simulation in the appropriate domain on an as‐needed basis. Should any error occur, the director could easily edit the states of any brains that have become aware of an anomaly before it spoils the simulation. Alternatively, the director could skip back a few seconds and rerun the simulation in a way that avoids the problem.” — Are You Living in a Computer Simulation?
Some people, like Elon Musk, simply try to live with the belief that they are living in a simulation. Maybe you should take seriously the possibility that you are living in some sort of computer simulation, but not that you are living in a simulation which tampers with your memories. But this is an ad hoc hypothesis. Once you admit that you are, or might be, living in a simulation, how could you possibly know that your memories aren’t being manipulated? It is much more principled to say that consciousness simply isn’t the kind of thing that could be digitally simulated at all, rather than to concede that you could be simulated but, for whatever reason, you somehow know that you’re not in one of those really bad simulations. Plausibly you know your own consciousness better than you know the distribution of simulations that might exist in the multiverse.
This doesn’t “prove” that consciousness is uncomputable. But it does suggest that the uncomputability of consciousness is a basic assumption of rationality that we need to make in order to stay sane. As the philosopher Brian Cutter suggested in an interview,
“If you start with the idea that it’s rationally justified to rely on priors that encode an anti-skeptical bias, that can get you to a kind of a priori justification for rejecting the possibility of computer-based consciousness.” — Cutter (2024, 50:45)
Divine simulations
We’re not living in a computer simulation. But what if we’re living in a different kind of simulation— a simulation created not by a finite, flawed alien, but by an infinite, perfect God? The “divine simulation hypothesis,” as it were, does not seem to suffer from the same problems as the computer simulation hypothesis, because God is supposed to be perfectly good by his nature. He cannot deceive nor be deceived, and therefore he would never put us in a nasty skeptical scenario. Quite the contrary: he seems to be the best possible protection against skeptical scenarios!
In an atheist universe, there’s no principled reason to think you’re not living in the Matrix, or in the Truman Show. In fact, physicists have shown that in very large, old universes, disembodied brains with humanlike experiences spontaneously pop into existence in outer space due to random chance much more often than humanlike beings evolve in ordinary solar systems. Given plausible assumptions about physics, most beings with experiences like yours are disembodied brains floating in outer space which will be destroyed in the next millisecond, or “Boltzmann brains,” rather than normal human beings. Since Boltzmann brains can’t trust their own memories, you can’t actually empirically rule out the possibility that you are a Boltzmann brain who popped into existence a fraction of a second ago!
If there is a benevolent God, however, we have an elegant, simple explanation for why we’re not Boltzmann brains, and why we’re not in the Matrix. God would also explain why computers can’t be conscious: God is the one who chooses which material objects are imbued with souls.15 He would never give a soul to a computer or a Boltzmann brain, because that would create a person in a skeptical scenario, and it would destroy basic principles of morality. If you’re interested in hearing more about this argument, I’d encourage you to check out this YouTube video by Ethan Muse and Pat Flynn:
Since we’ve established that simulations and books are similar in deep ways, it’s worth noting that Christian and Jewish philosophers often compare the relationship between God and his creation to the relationship between an author and a book. This analogy is so common that it has a name: it’s called the authorial analogy. It explains how God can know the entire future, and organize events in such a way to achieve his long-term goals. God doesn’t need to perform obvious miracles in order to get what he wants, because he is the Author of the whole story. Maybe he arranged the initial state of the universe in such a way that everything would play out exactly the way he wants. Or maybe he does intervene from time to time, but usually in ways that aren’t obvious.
The famous Christian writer C.S. Lewis used the authorial analogy extensively. Responding to the claim by Soviet cosmonaut Gherman Titov that he did not find God in outer space, Lewis wrote:
If God does exist, He is related to the universe more as an author is related to a play, than as one object in the universe is related to another. If God created the universe, he created spacetime, which is to the universe as the meter is to a poem, or the key is to music. To look for Him as one item within the framework which He himself invented is nonsensical.
— The Seeing Eye, 3:17
Earlier, Lewis used the authorial analogy to illustrate how God could have determined the outcome of the Battle of Dunkirk, a major battle of World War II, before creation:
Thus God must be supposed in predetermining the weather at Dunkirk to have taken fully into account the effect it would have not only on the destiny of two nations but (what is incomparably more important) on all the individuals involved on both sides, on all animals, vegetables and minerals within range, and finally on every atom in the universe. This may sound excessive, but in reality we are attributing to the Omniscient only an infinitely superior degree of the same kind of skill which a mere human novelist exercises daily in constructing his plot.
— Miracles, Appendix B
Lewis is using the real-world existence of authors and books to argue for the plausibility of a supreme Author of the universe, who is much more intelligent than any human author.
In the same way, the real-world existence of computer simulations makes it plausible that there is a supreme Simulator of the universe, who is much more powerful than any simulator in this world. Computer simulations show that it’s really possible for an intelligent being to create a highly detailed world that they fully control, whose future they can see in advance. They also make it plausible that God would “hide” from his creation most of the time, giving plausibly-deniable hints of his existence here and there, always leaving the possibility for people to doubt or reject him. Because most of the time, when we make simulated worlds, we don’t make it really obvious from the inside that there is a simulator controlling everything. The idea of God is now so plausible that the secular philosopher Nick Bostrom recently wrote:
“Human civilization is most likely not alone in the cosmos but is instead encompassed within a cosmic host.”
— AI Creation and the Cosmic Host (2024)
Bostrom uses the phrase “cosmic host” as a catch-all term to refer to the set of all superintelligent, super-powerful beings who are watching what humanity does. While he primarily focuses on the possibility that we are living in a computer simulation run by aliens, he does recognize that the cosmic host might include “supernatural beings, supported by some religious views” (page 3). Bostrom does not assume that the cosmic host has our best interests at heart, and in fact, his whole paper is premised on the assumption that they probably do not.
But the idea that the cosmic host is evil or indifferent creates problems for rational inquiry. If God is an alien who doesn’t care about us, we might have been created five seconds ago with false memories. All our friends might be unconscious non-player characters. The whole world might be destroyed tomorrow with no warning. You have to make some assumption about the goodness of God. And the simplest, least arbitrary assumption is that God is perfectly good. If everyone is in the business of making faith assumptions in order to stay sane, why not make the strongest, most optimistic assumption you possibly can? That’s basically what Christians are doing.
When I started drafting this essay in the spring of 2025, I was an agnostic. But as I thought about these issues, the existence of God seemed more and more plausible to me. About a year ago, I became a theist, but I was unsure which, if any, organized religion was actually true. I was surprised to discover that there is actually a lot of evidence for miracles performed by the Christian God in modern times. My friend Ethan Muse has written a few articles surveying the evidence for these miracles. You can read them on his Substack:
or watch his interview with Matt Fradd:
These miracles are just the tip of the iceberg. Other well-evidenced miracles include Our Lady of Zeitoun (here) and Our Lady of Lourdes (here). Skeptics must believe that all these miracles are somehow hoaxes and frauds, when the evidence points strongly in the opposite direction. In the face of this evidence, skepticism starts to look like conspiratorial thinking, not common sense.
God seems to like irony. Instead of overthrowing the Roman Empire violently, he sent his son to die on a cross, thereby kickstarting a movement that took over the Empire ideologically over the course of three centuries. It would also be ironic, in a similar way, for God to seed us with the scientific and technological ideas necessary to create fake gods, in the form of AIs, and fake heavens, in the form of virtual realities, so that we can realize, now at the end of history, that the gods and heavens that we can create in this world will never satisfy us. Only the true God and the true Heaven can do that.
Conclusion
We are not rejecting AI sentience because we’re insensitive, mean, or selfish. It’s because we recognize that the circle of moral concern has to stop somewhere in order for it to be meaningful at all. Ethics is about making distinctions between good and bad, real and fake, right and wrong. If AI can deserve moral consideration just by seeming conscious, we open ourselves up to a variety of paradoxes and absurdities, because we have dissolved the distinction between real consciousness and fake consciousness. The absurdity reaches its peak in the simulation argument, where we are supposed to take seriously the idea that the world we know and love is entirely fake— nothing but a video game played by an alien in the next universe up. This is conceivable, but that does nothing to show that it’s actually possible, let alone true.
Real sentience is grounded in biology. Organisms are irreversible, uncopyable, unique processes of individuation. We creatively repeat a lifecycle inherited from our ancestors, rather than replicating a digital sequence of instructions. As far as we know, the only materials that can support a process like this are carbon and water. While life elsewhere in the universe may use a liquid solvent other than water— some biologists have suggested ammonia as a possibility— it is hard to imagine that anything like a robot could evolve naturally. The distinction between life and nonlife seems to be real and universal, even if we cannot draw an infinitely sharp dividing line between the two. The impulse to dissolve a distinction, simply because it cannot be made perfectly precise, is nihilistic.
Unfortunately, artificial intelligence threatens to destroy the distinction between real and fake in almost every domain. AI-generated documents, photos, and videos have become nearly indistinguishable from real documents written by real people, real photos taken of real objects, and real videos recorded from real situations. This is already wreaking havoc in the education system, and it’s becoming increasingly difficult to trust anything you see online. The solution in each of these cases is to bring more of our lives back into the physical world, where the distinction between real and fake is easier to see. We shouldn’t completely shun technology. But we are probably already using it a bit too much.
In the next few decades, many people will give up on the search for real happiness, meaning, and fulfillment, and settle for the fake versions of these things, which will be easily accessible in the form of virtual reality and AI companions. If we legalize mind uploading, many of these people will eventually choose to upload, falling prey to the Siren song of digital immortality. An even larger number of people, while not succumbing to the temptation of fake happiness themselves, will be unwilling to judge others for choosing what’s fake over what’s real. But this attitude is a slippery slope toward the total collapse of the distinction between real and fake, and the victory of nihilism.
Tolerance is an important value. We shouldn’t use coercion to prevent people from living out most of their lives in VR, or from taking AI girlfriends instead of real ones. But there are some things we can do. Suicide is already illegal, and we definitely should not make an exception for mind uploading. AIs should not be given legal rights or personhood, under any circumstances. Governments should provide financial incentives for having real children, and not for AI “children.” And we should probably put rules in place to prevent automation in key areas, ensuring that humans are always in the loop of the economy and the government.
Automation will allow us to work less and play more, which is a good thing. But all play and no work is not a recipe for genuine happiness, either. Humans need to be needed, need to belong, need to make a difference. But this is impossible if we allow complete automation of every aspect of our lives. We will need to actively choose not to use certain kinds of technology. This will be difficult, but not impossible. We should choose which technologies to use and which to reject, based on how well they facilitate real relationships, real love, and real virtue.
Appendix: Quantum simulations
There’s one last argument for the impossibility of computer-based consciousness that I wanted to include in this essay, but I couldn’t figure out how to fit it in the main text. It centers on a famous paradox at the heart of quantum mechanics. I’ll try to make it as simple as possible, without getting into mathematical details.
Schrödinger’s cat is a famous thought experiment where we put a cat in a sealed box, and it is either killed or kept alive depending on the result of some random quantum event. Before we open the box, quantum theory says the cat is in a superposition of being alive and dead at the same time. Importantly, this does not mean the cat is either alive or dead, and we don’t know which.16 The scenario is interesting because there is a weird quantum effect called interference that happens when a system is in superposition, which doesn’t happen for “everyday” cases where we are simply ignorant of the state of a system. Traditionally, it’s believed that the cat is in an indeterminate state, which only gets “collapsed” into being alive or being dead when we open the box. But this is weird because we usually think of consciousness as an “absolute” thing— either the cat is experiencing being alive, or it is not.
In fact, we seem to be presupposing that consciousness is absolute when we say that the cat’s state is “collapsed” by a conscious human opening the box. Why is the human special? What if he himself is trapped inside a superposition in a larger box? That’s precisely what the Wigner’s friend thought experiment imagines— the scientist Wigner puts his friend Alice in a box, where she enters a superposition of observing an electron in the “spin up” state, and observing it in the “spin down” state. Now we can ask: when does Alice’s observation of the electron become real? When does Wigner’s observation of Alice’s observation become real? There are many attempts to answer this question in the literature. Famously, the many worlds interpretation says that there are two copies of Alice living in two different universes— one seeing spin up, the other seeing spin down— as well as two different copies of Wigner. But physicist Jacob Barandes points out that this creates all sorts of thorny paradoxes.
Ideally we would like a story of what happens in this scenario that allows us to believe in a single shared world, where facts about conscious observation are always objective and determinate. Strikingly, it turns out we can tell such a story, if we insist that consciousness is inherently irreversible. This is because superposition is only detectable in reversible systems.17 If the Wigner’s friend scenario were real, Wigner must have something like a “switch” he can flip to undo the superposition, thereby putting Alice back in her original state— much like you can start a new conversation with ChatGPT that forgets everything you just said to it.18 He would also be able to redo the superposition after undoing it, going back and forth as many times as he likes, before he opens the box and talks to Alice. What could it possibly be like to be Alice inside the box while this is happening? Scott Aaronson, a modern quantum computer expert, writes about this scenario:
“Notice in particular that, if the agent [Alice] could be manipulated in superposition, then as a direct byproduct of those manipulations, [Alice] would presumably undergo the same mental processes over and over, forwards in time as well as backwards in time… I hope I’m not alone in feeling a sense of vertigo about these questions! To me, it’s at least a plausible speculation that [Alice] doesn’t experience anything, and that the reasons why [she] doesn’t are related to [Wigner]’s very ability to manipulate [her] in these ways.”
— The Ghost in the Quantum Turing Machine, page 55
If consciousness is irreversible, Alice must be unconscious until the box is opened, and the absoluteness of observation is preserved. Problem solved. This isn’t a new idea, either— the pioneering quantum physicist Niels Bohr agreed that observation is tied to irreversibility. In a 1954 lecture, he explained:
“…every conscious experience corresponds to a residual impression in the organism, amounting to an irreversible recording in the nervous system of the outcome of processes which are not open to introspection…”
— Atomic Physics and Human Knowledge, page 77
In another lecture, he emphasized “the irreversibility characteristic of the very concept of observation” (page 89). If we deny that observation requires irreversibility, we would have to say that Alice is having a conscious experience that can be played in reverse and replicated over and over. Not only that, but this reversible consciousness can’t have any effect on what Wigner sees when he opens the box. Aaronson explains:
A crucial caveat is that, after the interference experiment was over, one would retain no reliable memories or publishable records about “what it was like”! For the very fact of such an experiment implies that one’s memories are being created and destroyed at will. Without the destruction of memories, we can’t get interference.
— The Ghost in the Quantum Turing Machine, footnote 45
Alice can never actually tell us what it’s like to be reversed and replayed over and over, just as a chatbot can’t tell us what it’s like to be reset and cloned. This should make you really start to question the idea that either one of them is conscious at all.
The analogy between Alice and the chatbot gets even stronger when you try to make the Wigner’s friend scenario more realistic. Aaronson points out that maintaining a person in superposition would require a lot more than a “sealed box.” This is because information about Alice would constantly leak out of the box in the form of infrared radiation. In order to prevent this, we would need sophisticated equipment to closely monitor and control everything going on inside Alice at the microscopic level, essentially turning Alice into a quantum computer. This explains the weirdness of the scenario. A real person cannot be “rewound” like a cassette tape, but a computer simulation of a person can be. The same reasoning applies to Schrödinger’s cat. Real cats can’t be brought back to life, but simulated cats can be. The paradoxes of Schrödinger’s cat and Wigner’s friend are resolved by rejecting the possibility of computer-based consciousness.
Could All Life Be Sentient?, page 2
These were specifically listed in The Cambridge Declaration on Consciousness, published in 2012 by a “prominent international group of cognitive neuroscientists, neuropharmacologists, neurophysiologists, neuroanatomists and computational neuroscientists.”
This phenomenon was apparently first described by H.V. Wilson in 1907. See Whole-Body Regeneration in Sponges: Diversity, Fine Mechanisms, and Future Prospects for a recent literature review on this topic.
See Chapter 13 of The Feeling of Life Itself (2020).
Hanson predicts that most ems will remember having had a lot of leisure time in the recent past, because this will make them more productive, even if leisure time accounts for a small fraction of the computing power dedicated to running ems (pg. 12). This is possible because one em can experience some leisure, then copy itself endlessly, endowing memories of leisure to all the resulting instances.
See Quantity of experience: brain-duplication and degrees of consciousness by Bostrom (2006). He doubled down on this idea more recently in Propositions Concerning Digital Minds and Society (2023).
The only known exception to this law is helium, which has a negative enthalpy of fusion.
Back of the envelope calculation suggests that a modern CPU has an entropy of about 0.6 J/K per gram, while a bacterium has an entropy of about 3 J/K per gram, assuming 70% water content.
For an in-depth discussion of this issue, see This review of Max Tegmark’s book also occurs infinitely often in the decimal expansion of π by Scott Aaronson.
I’m defending a form of externalism about mental content.
This is basically the view defended by Bergson in Matter and Memory (1896). Dreams and illusions are caused by the mind mixing together images drawn directly from the past.
For a small chatbot capable of producing fluent English (e.g. SmolLM2-135M), it would take roughly 4.2 person-years per token, assuming one operation per second using a hand calculator.
Musk hasn’t publicly said that he thinks the simulation is centered on him. But Nick Bostrom has publicly said that powerful, famous people like Elon Musk have much stronger reason to believe they are being simulated than the average person, since they are more interesting to simulate. The logic is obvious if you already take the simulation argument seriously.
In his original 2006 paper, he does not actually argue that we are likely living in a simulation, only that we are faced with a trilemma of possibilities, one of them being that we are living in a simulation. In his 2024 manuscript AI Creation and the Cosmic Host, however, he does state that we are likely living in a simulation, or at least something similar to one. In my estimation, he probably already believed the simulation hypothesis was true in 2006 or even earlier, but tried to present himself as an agnostic in public in order to sound more reasonable.
The final step in Bostrom’s argument requires an assumption, which he calls the “bland indifference” principle, that I actually think is invalid. But refuting this principle requires getting into very complex philosophical issues, and will have to wait for a later post. Denying the bland indifference principle ends up undermining the idea of computer-based consciousness in a different way.
Historically, Christian theologians have debated between traducianism, which states that the human soul is created by the parents at conception, and creationism, which states that God directly creates every human soul himself. The Catholic Church officially endorses creationism: ‘every spiritual soul is created immediately by God — it is not “produced” by the parents’ (CCC 336).
Hidden-variable theories say that there is a fact of the matter about the state of the cat, but this fact has no effect on the experimental outcome. Arguably, the cat “acts as if” it were in an indeterminate state, even if we stipulate that it is “truly” in a determinate state. If we say coherent superpositions can be conscious, then the hidden conscious state would be epiphenomenal— it would have no effect on experimental outcomes. Epiphenomenalism is usually seen as an unacceptable conclusion.
Closed systems evolve unitarily, and hence reversibly, until they are measured.
Aaronson et al. (2020) show this in On the Hardness of Detecting Macroscopic Superpositions.
Very nice post, I have argued against it here: https://thiagovscoelho.substack.com/p/response-to-taylor-belrose-on-ai
Thought-provoking!
One thing I was thinking about while reading is that the 4 key criteria you're laying out are quite entangled with each other. The reasons you lay out that the AIs we have now are not singular, private, ineffable, or qualitative all seem to stem from the fact that we can read out / copy their exact state. It seems to me that it's not too hard to make this impossible for us, though.
If I were to set up a machine with a locally running LLM, encrypt the disk & throw away the key, and maybe add a few analog components (like the person that hooked up a thermometer and light sensor to their copy of Claude and brought them to the eclipse!), suddenly we have an AI that ticks all your boxes. If you leave that system running long enough writing memory files and reading from its thermometer or whatever, you can no longer interact with it without irreversibly changing its state.
Your arguments against computers being able to host consciousness all seem to hinge on the fact that we can perfectly describe/copy them, but that's just an implementation detail, not inherent. More broadly, the motivation for why these specific criteria are necessary and sufficient is not really clear to me.