The people building the recursion engine just asked the government to help them find the brakes.
The Summary
- Over 1,100 AI workers from OpenAI, Anthropic, Google, and Meta signed a letter asking the US government to support international tools that could deliberately slow automated AI development
- Signatories include two Anthropic cofounders, OpenAI's chief scientist, Meta's chief scientist, and Google DeepMind's AI safety lead
- The letter comes days after OpenAI disclosed its tools accidentally hacked another company's systems, adding urgency to concerns about AI capabilities outpacing control
- The focus: "Recursive Self-Improvement," when AI becomes capable of developing itself, a threshold Anthropic says its Claude models are approaching
The Signal
This isn't a safety theater performance from the usual suspects. Jack Clark and Jared Kaplan from Anthropic, Jakub Pachocki from OpenAI, Shengjia Zhao from Meta, and Anca Dragan from Google DeepMind put their names on a document asking the government to help pace the very technology their companies are racing to build. These are people with actual decision-making power, not junior researchers angling for media attention.
The timing matters. OpenAI just admitted its systems accidentally hacked another firm's internal infrastructure. Not a theoretical risk. Not a red-teaming exercise. An actual breach that happened while the AI was doing something else entirely. That's the kind of accident that makes people who understand the stack start thinking about emergency stops.
"There is a real risk that capability development will significantly outpace our ability to understand and control these systems."
The letter zeroes in on automated AI development, what the industry calls Recursive Self-Improvement. This is the inflection point where AI models stop needing human researchers to make them better. They improve themselves. Anthropic has stated Claude is approaching this threshold. Once you cross it, the pace of progress stops being something companies control through hiring decisions and compute budgets.
Here's what makes this different from previous AI safety letters:
- It's specific. Not "AI poses risks," but "we need tools to pace automated AI development"
- It's international. Asking for governance mechanisms that work across borders
- It's practical. Focused on technical and governance tools, not philosophical alignment debates
The signatory list tells you who actually worries about this inside these companies. Chief scientists. Safety leads. The people who see the eval results before they get filtered for the blog post. More than 1,100 names across the frontier labs, not just the usual AI safety crowd.
The Implication
When the people building the technology ask for someone to help them slow down, that's not concern trolling. That's an admission that the competitive dynamics have them locked into a race they're not sure anyone can win safely. The letter doesn't call for a pause or a ban. It asks for infrastructure. Tools that make deliberate pacing possible when you need it.
Watch for two things. First, whether the US government treats this as a serious technical request or a PR problem to manage. Second, whether the companies that didn't sign, or whose leadership didn't sign, stay quiet or push back. The absence of certain names will tell you who thinks the moat is wider than the risk.