AI Cannot Outperform the Evidence It Was Built On
In 2019, we warned that automated systems trained on biased data would not correct inequality — they would cement it. We called it We Shape Our Tools, Thereafter Our Tools Shape Us, and we called for Affirmative Action for Algorithms.
Six years later, the evidence is in. The warning was not speculative. It was predictive.
Our new paper traces what happens when two of the systems most consequential to women’s lives — healthcare and criminal justice — are automated at scale using data that was never built with women in mind. Drawing on our recent research, Invisible by Design: Women’s Health as the Blind Spot in AI and Medicine and Gender Bias in Judicial Algorithms: A Global Analysis of Algorithmic Discrimination, we show that these aren’t separate crises. They share the same architecture of exclusion.
The pattern is the same in both systems. Historical exclusion of women from the foundational evidence base produces biased data. Biased data trains biased algorithms. Biased algorithms generate biased outcomes. And those outcomes feed back into the system as new “ground truth.” The loop closes. The bias locks in.
In medicine, the cascade starts with clinical research that defaulted to male physiology for most of the twentieth century. Women still make up only about 37% of clinical trial participants, and three-quarters of studies don’t report sex-stratified outcomes. From this foundation, diagnostic thresholds, drug dosing, and clinical guidelines were calibrated on the male body. When AI trains on electronic health records that encode those guidelines, the male default becomes the algorithmic default. The result: missed heart attacks, delayed diagnoses, adverse drug reactions in women at nearly twice the rate of men.
In criminal justice, algorithms learn from court decisions shaped by decades of gender stereotypes. Even when gender is removed as an explicit variable, discrimination routes through proxies — employment gaps that penalize caregiving, housing instability that reflects inequality, relationship histories that encode assumptions about dependence. The COMPAS system rated women “high risk” when their actual reoffending rate was less than half that of men rated the same. In the Netherlands, immigrant women were rated 40% higher risk than Dutch women with identical criminal histories.
The paper identifies four structural patterns that connect these systems: the automation of the credibility gap (where human skepticism toward women’s accounts becomes machine-level certainty); the mathematical impossibility of “neutral” algorithms; the missing variables problem (where the most consequential features of women’s lives — menstrual health, caregiving, experiences of gender-based violence — are invisible to models because no one built the infrastructure to capture them); and the failure of retrofitted fairness. You cannot bolt equity onto a biased foundation.
And the harm compounds along intersectional lines. Black women, Indigenous women, immigrant women, transgender women, disabled women — those already least visible in the data bear the system’s worst errors. A model that is “fair on average” can be deeply unfair for the communities that were already least served.
Healthcare is adopting AI 2.2 times faster than the rest of the economy. Courts across at least twenty U.S. states, multiple Canadian provinces, and pilot programs across Europe now use algorithmic tools for bail, sentencing, and case management. The window to intervene is narrowing.
What must change? The paper sets out an integrated agenda: rebuild the evidence base with sex-stratified research; redesign data infrastructure to capture what matters in women’s lives; mandate transparency and independent audits; enforce regulatory standards with teeth; include women in design — not as an afterthought, but from inception; and strengthen international cooperation through existing human rights mechanisms like CEDAW.
The technology is not destiny. These are human choices — what data to collect, what to prioritize, what to ignore. Better choices can change the course.
But once biased systems are embedded as infrastructure, once their outputs become training data for the next generation of models, the cost of correction rises and the likelihood of reform falls.
We are building the permanent architecture of automated decision-making right now. The question is whether women will be visible in it — or invisible by design.
Photo Credits: Deborah Lupton
https://betterimagesofai.org / https://creativecommons.org/licenses/by/4.0/