Applied AI · 2025

Thinking Outside the Bias

An evaluation framework for measuring bias in language-model answers and reasoning.

Research project with Coralie Sage and Finn Brunke

Python
LLM evaluation
BBQ
StereoSet

With Coralie Sage and Finn Brunke, I built a Python framework that separates the model answering a question from the model judging its bias. We tested reasoning models on BBQ and StereoSet, comparing prompting strategies and reasoning budgets.

Bias evaluation pipeline

Changing the judge changed the scores and sometimes the model rankings. Longer observed reasoning also correlated with higher judged bias, but changing the configured thinking budget barely changed reasoning length. These results depend on the chosen judge and benchmark.