The people building the intelligence are racing to make themselves obsolete — and betting they'll finish before they understand what they've built.
The Summary
- Jacob Coxon, 27-year-old ex-researcher from both Anthropic and OpenAI, testified before NYC Council that his job was literally "working to automate myself"
- OpenAI set internal milestones for automated AI researchers around 2027-2028, and they're on track or ahead of schedule — meaning months, not years
- Once AI can conduct its own research, humans will be "much further out of the loop than we are today, and today is already not great," according to someone who spent years inside the frontier labs
- Coxon warned NYC lawmakers that the AI industry is moving faster than it understands the risks
The Signal
This isn't speculation from a think tank. Coxon worked as a capabilities researcher — the people tasked with making AI more powerful, not safer. He saw the roadmaps. He knows what boxes they're trying to check and when. And the central box is self-improvement: AI that builds better AI without human hands on the wheel.
The timeline matters. When OpenAI set milestones for automated AI research at 2027-2028, that sounded distant. Now it's Q4 2026. Those milestones are 3-15 months out. Coxon says they're beating expectations.
"At my last job, I was, in some sense, working to automate myself."
What does automated AI research actually mean? It means the feedback loop tightens to the point where human researchers become spectators. Right now, a capabilities team designs experiments, interprets results, tweaks architectures, and pushes the next training run. Slow, expensive, bottlenecked by human intuition and sleep schedules. An automated researcher runs that loop at machine speed. It tries a thousand variations in the time it takes a human to try three. It doesn't need to explain its reasoning in a Slack thread.
Coxon told the NYC Council that the industry is racing ahead without fully understanding the risks. This is the part that cuts through the usual AI safety debates. He's not some outsider critic. He was inside Anthropic and OpenAI, companies that publicly position themselves as the "responsible" labs. If the people inside those buildings are saying they're moving too fast to understand what they're building, that's not ideology. That's geometry.
Key risks Coxon highlighted:
- Humans already struggle to interpret or control current AI systems
- Automated research removes the last human checkpoint in the development cycle
- Once machines are improving machines, oversight becomes retroactive at best
He resigned from Anthropic last month, making headlines as a whistleblower. Now he's testifying to city governments. That's the path of someone who tried internal channels and concluded they don't work. NYC is considering local AI safeguards, which means the policy conversation is fragmenting down to the municipal level because federal action remains stalled.
The Implication
If Coxon's timeline is right, we're about to see AI companies cross a threshold where their core product development happens faster than regulators, ethicists, or even their own leadership can track. Automated AI research isn't just a productivity unlock. It's the moment the thing being built starts building itself.
For anyone working in AI adjacent fields — data labeling, model evaluation, alignment research — the automation curve just steepened. The labs aren't hiding this. It's their stated primary goal. If you're betting your career on being in the loop, you need a new bet. The loop is shrinking.
Watch what NYC does next. Local regulation is clumsy and easy to route around, but it's also where things move when the federal government won't. If cities start imposing disclosure requirements or safety audits on AI deployments, the compliance landscape fractures. That creates openings for the first platforms that can prove their agents are auditable, interpretable, and human-steerable. The market for legible AI just got more valuable.