When your training data is three decades of racist policing, the algorithm doesn't fix bias—it automates it at scale.
The Summary
- A class action lawsuit in Ontario alleges an AI risk-assessment tool systematically places Black prisoners in maximum-security units at disproportionate rates
- The tool claims to objectively assess risk, but appears to encode historical bias from decades of discriminatory policing and sentencing data
- This isn't a bug. It's what happens when you train AI on a society's worst patterns and then give it real power over people's lives
The Signal
Ontario corrections has been using an AI system to classify prisoners by risk level, determining who goes to maximum security and who doesn't. The lawsuit argues the tool places Black inmates in max security facilities at rates that can't be explained by actual criminal history or behavior. The AI is doing exactly what it was trained to do: replicate the patterns in its training data.
Here's the problem. If you train an AI on 30 years of policing data from a system that disproportionately arrests, charges, and sentences Black people, the algorithm learns those patterns as truth. It sees correlations between race, zip code, prior interactions with police, and "risk." It doesn't understand that those correlations exist because of systemic racism. It just sees numbers.
"The algorithm doesn't question why the training data looks the way it does. It optimizes for it."
The Ontario case matters because it's happening in criminal justice, where the stakes are someone's freedom. But the same dynamic plays out everywhere AI touches hiring, lending, housing, insurance. An AI trained on biased historical data will produce biased future outcomes. The tool becomes a bias amplification engine.
And here's where the agent economy angle comes in. As AI agents take on more decision-making authority, the question of whose judgment they're replicating becomes existential. These aren't neutral math problems. They're encoded worldviews. If your AI agent is handling hiring decisions, underwriting loans, or triaging medical care, what patterns did it learn? From whose data? With what blind spots baked in?
The Implication
The lawsuit is a warning shot. AI agents will make millions of micro-decisions that shape people's lives, opportunities, and freedom. If we don't address bias in training data and model design now, we're not building an agent economy. We're building an automated caste system with plausible deniability.
For builders: audit your training data. Not just for accuracy, but for equity. For everyone else: when someone tells you an AI decision is "objective," ask what data it learned from. Algorithms don't eliminate human bias. They scale it.