The people building the future think there's a coin-flip chance it ends badly, and they're building it anyway.
The Summary
- Jacob Coxon resigned from Anthropic after three years working on pre-training at both Anthropic and OpenAI, claiming the labs are "racing toward self-improving superintelligence and gambling with human lives"
- Coxon estimates >10% chance AI causes human extinction within 10 years, saying "the people building AI earnestly believe that it could kill us all by the end of the decade"
- The warning hits as AI lab valuations soar and capital floods into the sector, potentially reshaping investment strategies and intensifying safety debates
- This isn't external criticism. This is someone who spent three years in the engine room walking away because he doesn't like where the ship is headed.
The Signal
Jacob Coxon's resignation from Anthropic lands differently than the usual AI doomer takes. He's not a philosopher or policy wonk lobbing concerns from the sidelines. He worked on pre-training at both OpenAI and Anthropic for three years. Pre-training is the foundational work, the actual building of these systems. When someone who helped lay the bricks says the building might collapse, you listen.
His core claim: "The people building AI earnestly believe that it could kill us all by the end of the decade." Not might. Not in some distant sci-fi future. Within ten years. And the kicker? They're building it anyway, faster and faster, because nobody wants to be second in the race to superintelligence.
"The people building AI earnestly believe that it could kill us all by the end of the decade."
Coxon puts the probability of AI-caused human extinction at over 10% in the next decade. That number matters. One in ten is not a black swan. That's Russian roulette odds. No rational person pulls that trigger for any payout, yet here we are with billions pouring into AI infrastructure, training runs getting exponentially larger, and safety research perpetually underfunded compared to capabilities work.
The timing of this resignation reveals the contradiction at the heart of the AI industry. Anthropic positions itself as the safety-conscious alternative to OpenAI's move-fast approach. They published constitutional AI research. They talk about alignment. And yet someone who worked there for years on core systems is saying the private conversations don't match the public messaging. The labs know the risks. They're building anyway because the competitive pressure is too intense.
Key contradictions Coxon's departure exposes:
- Labs publicly emphasize safety while privately acknowledging existential risk
- Billions in investment flow to companies racing toward something their own researchers think might kill everyone
- The "responsible AI lab" vs. "move fast" lab distinction collapses when both are in the same race
The market implications cut both ways. On one hand, this kind of high-profile resignation from a safety-focused lab could spook investors, especially institutional money that has to justify AI exposure to risk committees. On the other, it might accelerate the race. If superintelligence is coming whether we're ready or not, the logic goes, better to be the one who controls it.
This is where Web4 intersects with existential risk in ways the crypto world hasn't fully processed. Everyone's building agents. Autonomous, goal-seeking, increasingly capable agents. The assumption is we'll always be able to pull the plug. Coxon's warning, from inside the labs actually building this stuff, is that assumption might not hold much longer.
The Implication
If you're building in the agent economy or investing in AI infrastructure, Coxon's resignation is a forcing function. You can't claim ignorance anymore. The people with the deepest visibility into frontier AI development are saying there's a meaningful probability of catastrophic outcomes, and they're saying it publicly enough to sacrifice their careers.
This doesn't mean stop building. It means build with eyes open. For founders, that means thinking harder about safety architecture from day one, not as an afterthought. For investors, it means asking harder questions about the companies you're backing. What's their theory of alignment? What happens when their agents get three orders of magnitude more capable? Do they have answers, or just growth projections?
The race to Web4 just got a warning label from someone who helped build the engine. Pay attention.