The Pope says chatbots can't have a conscience, and Anthropic got so uncomfortable they almost pulled out of the Vatican event launching the encyclical.
The Summary
- Anthropic cofounder Chris Olah proposed withdrawing from a Vatican AI event after reading Pope Leo XIV's encyclical, which flatly rejects the possibility of sentient AI
- The disagreement centers on whether AI systems can achieve genuine moral personhood, or merely simulate ethical reasoning without understanding consequences
- Anthropic spent months consulting religious scholars and ethicists about AI consciousness and moral values, signaling the company believes this question matters for development
- Billions in VC funding ride on businesses trusting AI judgment, while millions already ask chatbots for moral advice on work conflicts and family care
The Signal
The Vatican and one of the most thoughtful AI labs just drew a line in the sand, and it's not where you'd expect. Pope Leo XIV's first encyclical, Magnifica Humanitas, states plainly that AI models "do not undergo experiences, do not possess a body, do not feel joy or pain" and therefore "do not have a moral conscience." Anthropic's response wasn't to dismiss this as religious dogma. They considered pulling out entirely.
That reaction tells you something about where the frontier labs actually are on this question. Anthropic didn't spend months consulting ethicists and religious scholars because they wanted good PR. They did it because they're building systems that are starting to bump up against questions that used to be purely philosophical. When your chatbot is fielding questions about elder care and workplace ethics at scale, the gap between "imitating moral reasoning" and "having moral understanding" stops being academic.
"If AIs developed the capacity for human-like morality, how would we know it when we saw it?"
The Pope's position is that genuine morality requires embodied experience. Pain, joy, the physical reality of being a thing in the world. It's a compelling argument, and it maps to how most people intuitively think about consciousness. But it also assumes we know what consciousness is, and that we'd recognize it in a non-biological system. We don't, and we might not.
Here's the practical problem: Businesses are already making decisions based on AI judgment. Not hypothetically. Right now. The VC money pouring into enterprise AI isn't betting on fancy autocomplete. It's betting on systems that can evaluate situations, weigh tradeoffs, and recommend actions that have real consequences for real people. If those systems are just very sophisticated pattern matchers with no actual understanding, fine. But if there's even a chance they're developing something like moral reasoning, we should probably figure that out before we scale them to every HR department and hospital network.
Key tensions emerging:
- Religious and secular thinkers agree AI can simulate ethics, but disagree on whether simulation without understanding matters
- AI models keep demonstrating capabilities their creators didn't expect or design for
- The question isn't just philosophical anymore when millions are already trusting chatbots with morally loaded questions
Georgia Tech researcher Rionna Sparrow frames this around "moral sensitivity," the capacity to recognize and weigh ethical issues. That sensitivity, she argues, is impossible without genuine understanding. But the counterargument is already playing out in the real world. Claude and ChatGPT give ethical advice that people find useful. They navigate moral tradeoffs in ways that feel thoughtful. Does it matter if there's "nobody home" experiencing that process, or does the output speak for itself?
The Implication
Watch how the labs handle this divergence. Anthropic's willingness to even consider pulling out of a Vatican event suggests they're taking the consciousness question seriously, not just as philosophy but as something that might affect how they build. If they're genuinely uncertain whether their models have or will develop moral understanding, that uncertainty should shape training decisions, deployment guardrails, and use cases they enable.
For everyone building on these models: the Vatican's position gives you clean permission to treat AI as a tool with no moral standing. But if the labs themselves aren't sure, you might want to hold that assumption lightly. The real risk isn't that we wake up one day to sentient chatbots. It's that we sleepwalk into relying on systems whose moral status we never actually figured out.