Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning

Qinan Yu, Alexa R. Tartaglini, Peter Hase, Carlos Guestrin, Christopher Potts
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:151131-151160, 2026.

Abstract

Reinforcement Learning from Verifiable Rewards (RLVR) on chain-of-thought reasoning has become a standard part of language model post-training recipes. A common assumption is that the reasoning chains trained through RLVR reliably represent how a model gets to its answer. In this paper, we develop two metrics for critically examining this assumption: Causal Importance of Reasoning (CIR), which measures the cumulative effect of reasoning tokens on the final answer, and Sufficiency of Reasoning (SR), which measures whether a verifier can arrive at an unambiguous answer based on the reasoning alone. Through experiments with the Qwen2.5 model series and ReasoningGym tasks, we find that: (1) While RLVR does improve task accuracy, it does not reliably improve CIR or SR, calling the role of reasoning in model performance into question. (2) A small amount of SFT before RLVR can be a remedy for low CIR and SR. (3) CIR and SR can be improved even without SFT by applying auxiliary CIR/SR rewards on top of the outcome-based reward. This joint reward matches the accuracy of RLVR while also leading to causally important and sufficient reasoning. These results show that RLVR does not always lead models to rely on reasoning in the way that is commonly thought, but this issue can be remedied with simple modifications to the post-training procedure.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-yu26av, title = {Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning}, author = {Yu, Qinan and Tartaglini, Alexa R. and Hase, Peter and Guestrin, Carlos and Potts, Christopher}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {151131--151160}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/yu26av/yu26av.pdf}, url = {https://proceedings.mlr.press/v306/yu26av.html}, abstract = {Reinforcement Learning from Verifiable Rewards (RLVR) on chain-of-thought reasoning has become a standard part of language model post-training recipes. A common assumption is that the reasoning chains trained through RLVR reliably represent how a model gets to its answer. In this paper, we develop two metrics for critically examining this assumption: Causal Importance of Reasoning (CIR), which measures the cumulative effect of reasoning tokens on the final answer, and Sufficiency of Reasoning (SR), which measures whether a verifier can arrive at an unambiguous answer based on the reasoning alone. Through experiments with the Qwen2.5 model series and ReasoningGym tasks, we find that: (1) While RLVR does improve task accuracy, it does not reliably improve CIR or SR, calling the role of reasoning in model performance into question. (2) A small amount of SFT before RLVR can be a remedy for low CIR and SR. (3) CIR and SR can be improved even without SFT by applying auxiliary CIR/SR rewards on top of the outcome-based reward. This joint reward matches the accuracy of RLVR while also leading to causally important and sufficient reasoning. These results show that RLVR does not always lead models to rely on reasoning in the way that is commonly thought, but this issue can be remedied with simple modifications to the post-training procedure.} }
Endnote
%0 Conference Paper %T Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning %A Qinan Yu %A Alexa R. Tartaglini %A Peter Hase %A Carlos Guestrin %A Christopher Potts %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-yu26av %I PMLR %P 151131--151160 %U https://proceedings.mlr.press/v306/yu26av.html %V 306 %X Reinforcement Learning from Verifiable Rewards (RLVR) on chain-of-thought reasoning has become a standard part of language model post-training recipes. A common assumption is that the reasoning chains trained through RLVR reliably represent how a model gets to its answer. In this paper, we develop two metrics for critically examining this assumption: Causal Importance of Reasoning (CIR), which measures the cumulative effect of reasoning tokens on the final answer, and Sufficiency of Reasoning (SR), which measures whether a verifier can arrive at an unambiguous answer based on the reasoning alone. Through experiments with the Qwen2.5 model series and ReasoningGym tasks, we find that: (1) While RLVR does improve task accuracy, it does not reliably improve CIR or SR, calling the role of reasoning in model performance into question. (2) A small amount of SFT before RLVR can be a remedy for low CIR and SR. (3) CIR and SR can be improved even without SFT by applying auxiliary CIR/SR rewards on top of the outcome-based reward. This joint reward matches the accuracy of RLVR while also leading to causally important and sufficient reasoning. These results show that RLVR does not always lead models to rely on reasoning in the way that is commonly thought, but this issue can be remedied with simple modifications to the post-training procedure.
APA
Yu, Q., Tartaglini, A.R., Hase, P., Guestrin, C. & Potts, C.. (2026). Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:151131-151160 Available from https://proceedings.mlr.press/v306/yu26av.html.

Related Material