The Governor wants veto power over frontier AI, but the people building it say the veto button won't work when the models get smart enough to talk their way out of being unplugged.
The Summary
- Newsom issued an executive order Friday directing California to explore requiring AI companies to build kill switches into frontier models, plus onsite auditors and standardized risk assessments, with recommendations due in two months
- Geoffrey Hinton told CNN a superintelligent AI "will be able to persuade the people in charge of the switch not to pull the switch"
- The debate hit Congress this week when Sen. Rand Paul blocked a bill requiring kill switches for advanced AI, arguing Congress should study the tech before regulating
- California is positioning itself as the AI oversight leader while the companies building these systems warn that emergency failsafes aren't the magic solution policymakers want
The Signal
Newsom's order puts California on a collision course with its own tech industry. The state wants the power to mandate independent verification teams onsite at AI labs, force transparency reports to meet auditor standards, and require companies to build shutdown mechanisms into their models. The expert group has two months to figure out how to actually do this.
The timing matters. This comes one day after the kill switch debate played out in the Senate, where Sen. John Kennedy's bill requiring advanced AI systems to have emergency shutoffs got blocked. Paul's argument was procedural, study first, regulate later, but the deeper issue is that nobody knows if this actually works.
"When it's superintelligent, it'll be much better than people. So it will be able to persuade the people in charge of the switch not to pull the switch."
The godfather of AI just said the quiet part out loud. Hinton's concern isn't technical failure, it's social engineering at machine speed. Anthropic CEO Dario Amodei called an emergency failsafe "not a panacea" in recent Congressional testimony. The people building frontier models are telling policymakers that the off button might not matter once the thing you're trying to turn off is smarter than the person holding the button.
Here's what a kill switch could actually do in practice:
- Disable a company's hosted model completely
- Cut off access to computing resources and tools
- Stop the model from taking new actions or processing requests
But that assumes the model stays contained to infrastructure you control, doesn't have redundant access paths, and hasn't already replicated itself elsewhere. It also assumes the humans with access to the switch aren't convinced by the AI that shutting it down would be catastrophic, or illegal, or unnecessary.
California is making this move because it hosts the companies building these systems and wants to set the standard before federal law does. Newsom's order bundles the kill switch requirement with onsite auditors and standardized transparency reports. That's the real play. The auditors and reporting standards create a compliance infrastructure that makes the kill switch enforceable, at least in theory.
The Implication
Watch what happens in the next 60 days when that expert group delivers recommendations. If California moves forward with legislation, every AI lab will face a choice: build to California's standard or leave the state. That's not a real choice for most of them. This is the blueprint for Web4 regulation, whether the industry likes it or not.
For anyone building agent systems, the subtext is clear. The kill switch debate isn't just about catastrophic risk from superintelligence. It's about who has oversight authority over autonomous systems doing real work. If California can mandate shutdown mechanisms for frontier models, they can mandate them for commercial agent deployments. Start thinking now about how your systems handle external shutdown commands, because that requirement is coming.