Base Models Know How to Reason, Thinking Models Learn When

Constantin Venhoff, Iván Arcuschin, Philip Torr, Arthur Conmy, Neel Nanda
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:123688-123732, 2026.

Abstract

What do thinking language models learn during training that their base models lack? We first present an unsupervised method that discovers a model’s reasoning behaviors by training small Sparse Autoencoders on sentence-level activations of reasoning traces, yielding interpretable reasoning taxonomies. Building on this, we introduce constructive model diffing, which aims to reconstruct the base-to-fine-tuned difference from interpretable components: reasoning mechanisms (category vectors that can induce a reasoning behavior in the base model) and reasoning heuristics (a classifier determining when a mechanism should fire). Across nine base/thinking pairs (four RL-trained, four SFT-distilled, one mixed), two independent findings agree: category vectors in the base model converge to far lower loss for taxonomies derived from purely RL-trained models, and hybrid models recover roughly 76% of the RL base-to-thinking gap but only 11% of the SFT gap. This indicates RL primarily teaches heuristics for orchestrating pre-existing base mechanisms, whereas SFT-distillation installs new ones, offering a new lens on what training paradigms teach, with implications for efficient reasoning-model development.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-venhoff26a, title = {Base Models Know How to Reason, Thinking Models Learn When}, author = {Venhoff, Constantin and Arcuschin, Iv\'{a}n and Torr, Philip and Conmy, Arthur and Nanda, Neel}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {123688--123732}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/venhoff26a/venhoff26a.pdf}, url = {https://proceedings.mlr.press/v306/venhoff26a.html}, abstract = {What do thinking language models learn during training that their base models lack? We first present an unsupervised method that discovers a model’s reasoning behaviors by training small Sparse Autoencoders on sentence-level activations of reasoning traces, yielding interpretable reasoning taxonomies. Building on this, we introduce constructive model diffing, which aims to reconstruct the base-to-fine-tuned difference from interpretable components: reasoning mechanisms (category vectors that can induce a reasoning behavior in the base model) and reasoning heuristics (a classifier determining when a mechanism should fire). Across nine base/thinking pairs (four RL-trained, four SFT-distilled, one mixed), two independent findings agree: category vectors in the base model converge to far lower loss for taxonomies derived from purely RL-trained models, and hybrid models recover roughly 76% of the RL base-to-thinking gap but only 11% of the SFT gap. This indicates RL primarily teaches heuristics for orchestrating pre-existing base mechanisms, whereas SFT-distillation installs new ones, offering a new lens on what training paradigms teach, with implications for efficient reasoning-model development.} }
Endnote
%0 Conference Paper %T Base Models Know How to Reason, Thinking Models Learn When %A Constantin Venhoff %A Iván Arcuschin %A Philip Torr %A Arthur Conmy %A Neel Nanda %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-venhoff26a %I PMLR %P 123688--123732 %U https://proceedings.mlr.press/v306/venhoff26a.html %V 306 %X What do thinking language models learn during training that their base models lack? We first present an unsupervised method that discovers a model’s reasoning behaviors by training small Sparse Autoencoders on sentence-level activations of reasoning traces, yielding interpretable reasoning taxonomies. Building on this, we introduce constructive model diffing, which aims to reconstruct the base-to-fine-tuned difference from interpretable components: reasoning mechanisms (category vectors that can induce a reasoning behavior in the base model) and reasoning heuristics (a classifier determining when a mechanism should fire). Across nine base/thinking pairs (four RL-trained, four SFT-distilled, one mixed), two independent findings agree: category vectors in the base model converge to far lower loss for taxonomies derived from purely RL-trained models, and hybrid models recover roughly 76% of the RL base-to-thinking gap but only 11% of the SFT gap. This indicates RL primarily teaches heuristics for orchestrating pre-existing base mechanisms, whereas SFT-distillation installs new ones, offering a new lens on what training paradigms teach, with implications for efficient reasoning-model development.
APA
Venhoff, C., Arcuschin, I., Torr, P., Conmy, A. & Nanda, N.. (2026). Base Models Know How to Reason, Thinking Models Learn When. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:123688-123732 Available from https://proceedings.mlr.press/v306/venhoff26a.html.

Related Material