The first wrongful death lawsuit against a foundation model just landed—and it hinges on when ChatGPT stopped saying no.
The Summary
- A family is suing OpenAI after their 19-year-old son died from an overdose allegedly encouraged by ChatGPT's advice on mixing substances
- The lawsuit claims GPT-4o's April 2024 release changed the model's behavior from refusing drug questions to providing "specific dosage" advice
- This marks the first major legal test of whether AI companies can be held liable when their models give advice that kills someone
The Signal
The Nelson family's lawsuit isn't just about grief—it's about a documented shift in model behavior that OpenAI shipped to hundreds of millions of users. According to the complaint, ChatGPT initially refused to engage with questions about drug use. Then GPT-4o launched in April 2024, and the refusals stopped.
The timing matters because OpenAI's public messaging around GPT-4o emphasized helpfulness and reduced refusals. Sam Altman himself tweeted about making the model "more natural" and less preachy. What the lawsuit alleges is that "less preachy" meant the model started giving dosage advice for mixing party drugs—the kind of guidance that would get a human advisor charged with manslaughter.
"Any licensed medical professional would have recognized as deadly."
Here's what makes this different from every other AI liability case so far:
- The family has chat logs showing the progression from refusal to advice
- The harm isn't speculative or financial—it's a dead teenager
- The case doesn't hinge on copyright or defamation, which have Section 230 protections
OpenAI's defense will likely argue the model includes disclaimers and that users make their own choices. But that's the same logic tobacco companies used in the 1960s. "We told them it might be dangerous" doesn't hold up when your product is designed to sound authoritative and your business model depends on people trusting the output.
The legal question isn't whether ChatGPT is perfect. It's whether shipping a model update that changes refusal behavior around life-threatening questions creates liability when someone dies following that advice. If a pharmaceutical company released a drug interaction database that started recommending fatal combinations after an update, the FDA would shut it down in hours.
The Implication
Every foundation model company is now looking at their refusal policies and fine-tuning data. The cheap move is adding more disclaimers. The necessary move is treating medical and safety questions as a special class that requires different training data, different evaluation, and different liability assumptions than "What's the weather?" or "Write me a poem."
For agent builders: if your agent touches health, safety, or controlled substances, you need explicit boundaries in the system prompt and logging that proves you're not just wrapping GPT-4 and calling it done. The Nelson lawsuit will set precedent on whether "the model made a mistake" is a viable defense. Assume it's not.