AI & Equality
What Does Accuracy Actually Measure? The Hidden Geometry of AI Failure
“Show Me the Data” <the_benchmark_problem> Series with Leyla Roksan Caglar
November 4th | 3PM GMT
“Show Me the Data” <the_benchmark_problem> is a new AI & Equality series on the hidden assumptions built into AI benchmarks. Each session takes one influential benchmark or evaluation framework – in vision, language, healthcare, criminal justice – to ask who designed it, what population was it validated on, what does it reward, and who does it fail?
We are delighted to open the series with its curator, (Icahn School of Medicine at Mount Sinai), who will lead the first session: What Does Accuracy Actually Measure? The Hidden Geometry of AI Failure
About the session:
When we say an AI system performs well, we almost always mean it scores well on a benchmark. But benchmark accuracy is structurally blind to some of the most important things about how a system fails – including failures that fall disproportionately on specific populations. Drawing on her own research comparing how humans and deep neural networks compress and generalize visual information, this talk makes the case that the benchmark problem is not a matter of imprecision but of kind: accuracy-based evaluation cannot recover the geometry of how a system fails, who gets collapsed into dominant error categories, or whether a system’s compression strategy resembles biological vision or something fundamentally more brittle.
Leyla will introduce rate-distortion geometry as both a diagnostic and an accountability tool – one that can be applied to deployed systems from behavioral data alone, without access to model internals – and use it to reframe what responsible AI evaluation would actually need to measure.
Leyla Roksan Caglar is a Postdoctoral Fellow in the Windreich Department for AI & Human Health at the Icahn School of Medicine at Mount Sinai, working at the intersection of neuroscience, AI, and cognitive science to understand how systems with different learning mechanisms acquire, generalize, and represent perceptual information and how we can use these insights to build safe, transparent, and responsible AI systems.
Last modified: October 27, 2026