Discovering Interpretable Algorithms by Decompiling Transformers to RASP

Xinting Huang, Aleksandra Bakalova, Satwik Bhattamishra, William Merrill, Michael Hahn
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:46309-46412, 2026.

Abstract

Recent work has shown that the computations of Transformers can be simulated in the RASP family of programming languages. These findings have enabled improved understanding of the expressive capacity and generalization abilities of Transformers. In particular, Transformers have been suggested to length-generalize exactly on problems that have simple RASP programs. However, it remains open whether trained models actually implement simple interpretable programs. In this paper, we present a general method to extract such programs from trained Transformers. The idea is to faithfully re-parameterize a Transformer as a RASP program and then apply causal interventions to discover a small sufficient sub-program. In experiments on small Transformers trained on algorithmic and formal language tasks, we show that our method often recovers simple and interpretable RASP programs from length-generalizing transformers. Our results provide the most direct evidence so far that Transformers internally implement simple RASP programs.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-huang26b, title = {Discovering Interpretable Algorithms by Decompiling Transformers to {RASP}}, author = {Huang, Xinting and Bakalova, Aleksandra and Bhattamishra, Satwik and Merrill, William and Hahn, Michael}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {46309--46412}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/huang26b/huang26b.pdf}, url = {https://proceedings.mlr.press/v306/huang26b.html}, abstract = {Recent work has shown that the computations of Transformers can be simulated in the RASP family of programming languages. These findings have enabled improved understanding of the expressive capacity and generalization abilities of Transformers. In particular, Transformers have been suggested to length-generalize exactly on problems that have simple RASP programs. However, it remains open whether trained models actually implement simple interpretable programs. In this paper, we present a general method to extract such programs from trained Transformers. The idea is to faithfully re-parameterize a Transformer as a RASP program and then apply causal interventions to discover a small sufficient sub-program. In experiments on small Transformers trained on algorithmic and formal language tasks, we show that our method often recovers simple and interpretable RASP programs from length-generalizing transformers. Our results provide the most direct evidence so far that Transformers internally implement simple RASP programs.} }
Endnote
%0 Conference Paper %T Discovering Interpretable Algorithms by Decompiling Transformers to RASP %A Xinting Huang %A Aleksandra Bakalova %A Satwik Bhattamishra %A William Merrill %A Michael Hahn %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-huang26b %I PMLR %P 46309--46412 %U https://proceedings.mlr.press/v306/huang26b.html %V 306 %X Recent work has shown that the computations of Transformers can be simulated in the RASP family of programming languages. These findings have enabled improved understanding of the expressive capacity and generalization abilities of Transformers. In particular, Transformers have been suggested to length-generalize exactly on problems that have simple RASP programs. However, it remains open whether trained models actually implement simple interpretable programs. In this paper, we present a general method to extract such programs from trained Transformers. The idea is to faithfully re-parameterize a Transformer as a RASP program and then apply causal interventions to discover a small sufficient sub-program. In experiments on small Transformers trained on algorithmic and formal language tasks, we show that our method often recovers simple and interpretable RASP programs from length-generalizing transformers. Our results provide the most direct evidence so far that Transformers internally implement simple RASP programs.
APA
Huang, X., Bakalova, A., Bhattamishra, S., Merrill, W. & Hahn, M.. (2026). Discovering Interpretable Algorithms by Decompiling Transformers to RASP. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:46309-46412 Available from https://proceedings.mlr.press/v306/huang26b.html.

Related Material