Reading Between the Tokens: Improving Preference Predictions through Mechanistic Forecasting

Sarah Ball, Simeon Allmendinger, Frauke Kreuter, Niklas Kühl
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:6002-6023, 2026.

Abstract

Large language models are increasingly used to predict human preferences in both scientific and business endeavors, yet current approaches rely exclusively on analyzing model outputs without considering the underlying mechanisms. Using election forecasting as a test case, we introduce mechanistic forecasting, a method that demonstrates that probing internal model representations offers a fundamentally different—and sometimes more effective— approach to preference prediction. Examining over 24 million configurations across 7 models, 6 national elections, multiple persona attributes, and prompt variations, we systematically analyze how demographic and ideological information activates latent party-encoding components within the respective models. We find that leveraging this internal knowledge via mechanistic forecasting, opposed to solely relying on surface-level predictions, can improve prediction accuracy. The effects vary across demographic versus opinion-based attributes, political parties, national contexts, and models. Our findings demonstrate that the latent representational structure of LLMs contains systematic, exploitable information about human preferences, establishing a new paradigm for using language models in social science prediction tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-ball26a, title = {Reading Between the Tokens: Improving Preference Predictions through Mechanistic Forecasting}, author = {Ball, Sarah and Allmendinger, Simeon and Kreuter, Frauke and K\"{u}hl, Niklas}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {6002--6023}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/ball26a/ball26a.pdf}, url = {https://proceedings.mlr.press/v306/ball26a.html}, abstract = {Large language models are increasingly used to predict human preferences in both scientific and business endeavors, yet current approaches rely exclusively on analyzing model outputs without considering the underlying mechanisms. Using election forecasting as a test case, we introduce mechanistic forecasting, a method that demonstrates that probing internal model representations offers a fundamentally different—and sometimes more effective— approach to preference prediction. Examining over 24 million configurations across 7 models, 6 national elections, multiple persona attributes, and prompt variations, we systematically analyze how demographic and ideological information activates latent party-encoding components within the respective models. We find that leveraging this internal knowledge via mechanistic forecasting, opposed to solely relying on surface-level predictions, can improve prediction accuracy. The effects vary across demographic versus opinion-based attributes, political parties, national contexts, and models. Our findings demonstrate that the latent representational structure of LLMs contains systematic, exploitable information about human preferences, establishing a new paradigm for using language models in social science prediction tasks.} }
Endnote
%0 Conference Paper %T Reading Between the Tokens: Improving Preference Predictions through Mechanistic Forecasting %A Sarah Ball %A Simeon Allmendinger %A Frauke Kreuter %A Niklas Kühl %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-ball26a %I PMLR %P 6002--6023 %U https://proceedings.mlr.press/v306/ball26a.html %V 306 %X Large language models are increasingly used to predict human preferences in both scientific and business endeavors, yet current approaches rely exclusively on analyzing model outputs without considering the underlying mechanisms. Using election forecasting as a test case, we introduce mechanistic forecasting, a method that demonstrates that probing internal model representations offers a fundamentally different—and sometimes more effective— approach to preference prediction. Examining over 24 million configurations across 7 models, 6 national elections, multiple persona attributes, and prompt variations, we systematically analyze how demographic and ideological information activates latent party-encoding components within the respective models. We find that leveraging this internal knowledge via mechanistic forecasting, opposed to solely relying on surface-level predictions, can improve prediction accuracy. The effects vary across demographic versus opinion-based attributes, political parties, national contexts, and models. Our findings demonstrate that the latent representational structure of LLMs contains systematic, exploitable information about human preferences, establishing a new paradigm for using language models in social science prediction tasks.
APA
Ball, S., Allmendinger, S., Kreuter, F. & Kühl, N.. (2026). Reading Between the Tokens: Improving Preference Predictions through Mechanistic Forecasting. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:6002-6023 Available from https://proceedings.mlr.press/v306/ball26a.html.

Related Material