Mechanics of Bias and Reasoning: Interpreting the Impact of Chain-of-Thought Prompting on Gender Bias in LLMs

Edie Pearman, Sophia Osborne, Mira Kandlikar-Bloch, Mina Arzaghi, Florian Carichon, Golnoosh Farnadi
Proceedings of The Algorithmic Fairness Across Alignment Procedures and Agentic Systems, PMLR 344:1-33, 2026.

Abstract

Large language models (LLMs) are increasingly deployed in socially sensitive settings despite substantial documentation that they encode gender biases. Chain-of-Thought (CoT) prompting has been proposed as a bias-mitigation approach. However, existing evaluations primarily focus on changes in LLM benchmark performance, providing limited insight into whether apparent bias reductions reflect meaningful changes in a model’s internal mechanisms. In this work, we investigate how CoT prompting affects gender bias in LLMs, combining benchmark-based evaluation with mechanistic interpretability techniques and reasoning chain failure analysis. Our results confirm a stereotypical bias present in LLM outputs across benchmarks, showing that CoT prompting does not consistently reduce the bias gap. Mechanistic analyses reveal that although CoT balances biased behavior in certain attention head clusters, gender bias remains embedded in hidden representations, indicating only superficial mitigation. Inspection of reasoning chains further suggests that these improvements frequently co-occur with bias statements, and may reflect dataset familiarity rather than genuine bias mitigation.

Cite this Paper

BibTeX
@InProceedings{pmlr-v344-pearman26a, title = {Mechanics of Bias and Reasoning: Interpreting the Impact of Chain-of-Thought Prompting on Gender Bias in LLMs}, author = {Pearman, Edie and Osborne, Sophia and Kandlikar-Bloch, Mira and Arzaghi, Mina and Carichon, Florian and Farnadi, Golnoosh}, booktitle = {Proceedings of The Algorithmic Fairness Across Alignment Procedures and Agentic Systems}, pages = {1--33}, year = {2026}, editor = {Tang, Zeyu and Ganesh, Prakhar and Dieng, Awa and Rateike, Miriam and Watson-Daniels, Jamelle and Farnadi, Golnoosh}, volume = {344}, series = {Proceedings of Machine Learning Research}, month = {26 Apr}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v344/main/assets/pearman26a/pearman26a.pdf}, url = {https://proceedings.mlr.press/v344/pearman26a.html}, abstract = {Large language models (LLMs) are increasingly deployed in socially sensitive settings despite substantial documentation that they encode gender biases. Chain-of-Thought (CoT) prompting has been proposed as a bias-mitigation approach. However, existing evaluations primarily focus on changes in LLM benchmark performance, providing limited insight into whether apparent bias reductions reflect meaningful changes in a model’s internal mechanisms. In this work, we investigate how CoT prompting affects gender bias in LLMs, combining benchmark-based evaluation with mechanistic interpretability techniques and reasoning chain failure analysis. Our results confirm a stereotypical bias present in LLM outputs across benchmarks, showing that CoT prompting does not consistently reduce the bias gap. Mechanistic analyses reveal that although CoT balances biased behavior in certain attention head clusters, gender bias remains embedded in hidden representations, indicating only superficial mitigation. Inspection of reasoning chains further suggests that these improvements frequently co-occur with bias statements, and may reflect dataset familiarity rather than genuine bias mitigation.} }
Endnote
%0 Conference Paper %T Mechanics of Bias and Reasoning: Interpreting the Impact of Chain-of-Thought Prompting on Gender Bias in LLMs %A Edie Pearman %A Sophia Osborne %A Mira Kandlikar-Bloch %A Mina Arzaghi %A Florian Carichon %A Golnoosh Farnadi %B Proceedings of The Algorithmic Fairness Across Alignment Procedures and Agentic Systems %C Proceedings of Machine Learning Research %D 2026 %E Zeyu Tang %E Prakhar Ganesh %E Awa Dieng %E Miriam Rateike %E Jamelle Watson-Daniels %E Golnoosh Farnadi %F pmlr-v344-pearman26a %I PMLR %P 1--33 %U https://proceedings.mlr.press/v344/pearman26a.html %V 344 %X Large language models (LLMs) are increasingly deployed in socially sensitive settings despite substantial documentation that they encode gender biases. Chain-of-Thought (CoT) prompting has been proposed as a bias-mitigation approach. However, existing evaluations primarily focus on changes in LLM benchmark performance, providing limited insight into whether apparent bias reductions reflect meaningful changes in a model’s internal mechanisms. In this work, we investigate how CoT prompting affects gender bias in LLMs, combining benchmark-based evaluation with mechanistic interpretability techniques and reasoning chain failure analysis. Our results confirm a stereotypical bias present in LLM outputs across benchmarks, showing that CoT prompting does not consistently reduce the bias gap. Mechanistic analyses reveal that although CoT balances biased behavior in certain attention head clusters, gender bias remains embedded in hidden representations, indicating only superficial mitigation. Inspection of reasoning chains further suggests that these improvements frequently co-occur with bias statements, and may reflect dataset familiarity rather than genuine bias mitigation.
APA
Pearman, E., Osborne, S., Kandlikar-Bloch, M., Arzaghi, M., Carichon, F. & Farnadi, G.. (2026). Mechanics of Bias and Reasoning: Interpreting the Impact of Chain-of-Thought Prompting on Gender Bias in LLMs. Proceedings of The Algorithmic Fairness Across Alignment Procedures and Agentic Systems, in Proceedings of Machine Learning Research 344:1-33 Available from https://proceedings.mlr.press/v344/pearman26a.html.

Related Material