Compositional Causal Reasoning Evaluation in Language Models

Jacqueline R. M. A. Maasch, Alihan Hüyük, Xinnuo Xu, Aditya V. Nori, Javier Gonzalez
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:42325-42365, 2025.

Abstract

Causal reasoning and compositional reasoning are two core aspirations in AI. Measuring the extent of these behaviors requires principled evaluation methods. We explore a unified perspective that considers both behaviors simultaneously, termed compositional causal reasoning (CCR): the ability to infer how causal measures compose and, equivalently, how causal quantities propagate through graphs. We instantiate a framework for the systematic evaluation of CCR for the average treatment effect and the probability of necessity and sufficiency. As proof of concept, we demonstrate CCR evaluation for language models in the Llama, Phi, and GPT families. On a math word problem, our framework revealed a range of taxonomically distinct error patterns. CCR errors increased with the complexity of causal paths for all models except o1.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-maasch25a, title = {Compositional Causal Reasoning Evaluation in Language Models}, author = {Maasch, Jacqueline R. M. A. and H\"{u}y\"{u}k, Alihan and Xu, Xinnuo and Nori, Aditya V. and Gonzalez, Javier}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {42325--42365}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/maasch25a/maasch25a.pdf}, url = {https://proceedings.mlr.press/v267/maasch25a.html}, abstract = {Causal reasoning and compositional reasoning are two core aspirations in AI. Measuring the extent of these behaviors requires principled evaluation methods. We explore a unified perspective that considers both behaviors simultaneously, termed compositional causal reasoning (CCR): the ability to infer how causal measures compose and, equivalently, how causal quantities propagate through graphs. We instantiate a framework for the systematic evaluation of CCR for the average treatment effect and the probability of necessity and sufficiency. As proof of concept, we demonstrate CCR evaluation for language models in the Llama, Phi, and GPT families. On a math word problem, our framework revealed a range of taxonomically distinct error patterns. CCR errors increased with the complexity of causal paths for all models except o1.} }
Endnote
%0 Conference Paper %T Compositional Causal Reasoning Evaluation in Language Models %A Jacqueline R. M. A. Maasch %A Alihan Hüyük %A Xinnuo Xu %A Aditya V. Nori %A Javier Gonzalez %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-maasch25a %I PMLR %P 42325--42365 %U https://proceedings.mlr.press/v267/maasch25a.html %V 267 %X Causal reasoning and compositional reasoning are two core aspirations in AI. Measuring the extent of these behaviors requires principled evaluation methods. We explore a unified perspective that considers both behaviors simultaneously, termed compositional causal reasoning (CCR): the ability to infer how causal measures compose and, equivalently, how causal quantities propagate through graphs. We instantiate a framework for the systematic evaluation of CCR for the average treatment effect and the probability of necessity and sufficiency. As proof of concept, we demonstrate CCR evaluation for language models in the Llama, Phi, and GPT families. On a math word problem, our framework revealed a range of taxonomically distinct error patterns. CCR errors increased with the complexity of causal paths for all models except o1.
APA
Maasch, J.R.M.A., Hüyük, A., Xu, X., Nori, A.V. & Gonzalez, J.. (2025). Compositional Causal Reasoning Evaluation in Language Models. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:42325-42365 Available from https://proceedings.mlr.press/v267/maasch25a.html.

Related Material