Predicting the Emergence of Induction Heads in Language Model Pretraining

Tatsuya Aoyama, Ethan Gotlieb Wilcox, Nathan Schneider
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:3164-3193, 2026.

Abstract

Specialized attention heads dubbed induction heads (IHs) have been argued to underlie the remarkable in-context learning capabilities of modern language models; yet, a precise characterization of their emergence, especially in the context of language modeling, remains wanting. In this study, we investigate the relationship between statistical properties of the training data and IH formation in both natural and synthetic training data settings. We show that: (1) a simple equation combining batch size and context size predicts the point at which IHs form and that this emergence point is agnostic to model size; (2) surface bigram repetition frequency and reliability strongly affect the formation of IHs, and we find an effective decision boundary in terms of these two values; (3) local dependency with high bigram repetition frequency and reliability is sufficient for IH formation, but categoriality and the shape of the marginal distribution appear to modulate IH formation near the decision boundary.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-aoyama26a, title = {Predicting the Emergence of Induction Heads in Language Model Pretraining}, author = {Aoyama, Tatsuya and Wilcox, Ethan Gotlieb and Schneider, Nathan}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {3164--3193}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/aoyama26a/aoyama26a.pdf}, url = {https://proceedings.mlr.press/v306/aoyama26a.html}, abstract = {Specialized attention heads dubbed induction heads (IHs) have been argued to underlie the remarkable in-context learning capabilities of modern language models; yet, a precise characterization of their emergence, especially in the context of language modeling, remains wanting. In this study, we investigate the relationship between statistical properties of the training data and IH formation in both natural and synthetic training data settings. We show that: (1) a simple equation combining batch size and context size predicts the point at which IHs form and that this emergence point is agnostic to model size; (2) surface bigram repetition frequency and reliability strongly affect the formation of IHs, and we find an effective decision boundary in terms of these two values; (3) local dependency with high bigram repetition frequency and reliability is sufficient for IH formation, but categoriality and the shape of the marginal distribution appear to modulate IH formation near the decision boundary.} }
Endnote
%0 Conference Paper %T Predicting the Emergence of Induction Heads in Language Model Pretraining %A Tatsuya Aoyama %A Ethan Gotlieb Wilcox %A Nathan Schneider %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-aoyama26a %I PMLR %P 3164--3193 %U https://proceedings.mlr.press/v306/aoyama26a.html %V 306 %X Specialized attention heads dubbed induction heads (IHs) have been argued to underlie the remarkable in-context learning capabilities of modern language models; yet, a precise characterization of their emergence, especially in the context of language modeling, remains wanting. In this study, we investigate the relationship between statistical properties of the training data and IH formation in both natural and synthetic training data settings. We show that: (1) a simple equation combining batch size and context size predicts the point at which IHs form and that this emergence point is agnostic to model size; (2) surface bigram repetition frequency and reliability strongly affect the formation of IHs, and we find an effective decision boundary in terms of these two values; (3) local dependency with high bigram repetition frequency and reliability is sufficient for IH formation, but categoriality and the shape of the marginal distribution appear to modulate IH formation near the decision boundary.
APA
Aoyama, T., Wilcox, E.G. & Schneider, N.. (2026). Predicting the Emergence of Induction Heads in Language Model Pretraining. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:3164-3193 Available from https://proceedings.mlr.press/v306/aoyama26a.html.

Related Material