How Few-Shot Examples Add Up: A Causal Decomposition of Function Vectors in In-Context Learning

Entang Wang, Yiwei Wang, Aleksandra Bakalova, Michael Hahn
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:130289-130358, 2026.

Abstract

In-context learning (ICL) excels at new tasks from minimal examples, yet we still lack a mechanistic explanation of how few-shot prompts shape a model’s function vector (FV)–a causal activation direction that drives task behavior on the ICL query. Across tasks and models, an $n$-shot FV is well-approximated by a linear combination of example-level sub-FVs, suggesting additive and composable contributions from individual demonstrations. Beyond additivity, we show that models contextualize individual examples’ representations based on prior examples to adaptively reweight which demonstrations dominate the FV: attention shifts toward examples that are more informative and less ambiguous under the context. Finally, a causal decomposition separates Query–Key routing from Value updates, finding that contextualization’s most consistent contributions to FV quality arise from Query–Key alignment–particularly in ambiguous settings–while Value-mediated effects are more heterogeneous. Together, these results unify additive superposition with context-dependent attention reweighting into a mechanistic, testable account of how few-shot prompts implement tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-wang26hp, title = {How Few-Shot Examples Add Up: A Causal Decomposition of Function Vectors in In-Context Learning}, author = {Wang, Entang and Wang, Yiwei and Bakalova, Aleksandra and Hahn, Michael}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {130289--130358}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/wang26hp/wang26hp.pdf}, url = {https://proceedings.mlr.press/v306/wang26hp.html}, abstract = {In-context learning (ICL) excels at new tasks from minimal examples, yet we still lack a mechanistic explanation of how few-shot prompts shape a model’s function vector (FV)–a causal activation direction that drives task behavior on the ICL query. Across tasks and models, an $n$-shot FV is well-approximated by a linear combination of example-level sub-FVs, suggesting additive and composable contributions from individual demonstrations. Beyond additivity, we show that models contextualize individual examples’ representations based on prior examples to adaptively reweight which demonstrations dominate the FV: attention shifts toward examples that are more informative and less ambiguous under the context. Finally, a causal decomposition separates Query–Key routing from Value updates, finding that contextualization’s most consistent contributions to FV quality arise from Query–Key alignment–particularly in ambiguous settings–while Value-mediated effects are more heterogeneous. Together, these results unify additive superposition with context-dependent attention reweighting into a mechanistic, testable account of how few-shot prompts implement tasks.} }
Endnote
%0 Conference Paper %T How Few-Shot Examples Add Up: A Causal Decomposition of Function Vectors in In-Context Learning %A Entang Wang %A Yiwei Wang %A Aleksandra Bakalova %A Michael Hahn %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-wang26hp %I PMLR %P 130289--130358 %U https://proceedings.mlr.press/v306/wang26hp.html %V 306 %X In-context learning (ICL) excels at new tasks from minimal examples, yet we still lack a mechanistic explanation of how few-shot prompts shape a model’s function vector (FV)–a causal activation direction that drives task behavior on the ICL query. Across tasks and models, an $n$-shot FV is well-approximated by a linear combination of example-level sub-FVs, suggesting additive and composable contributions from individual demonstrations. Beyond additivity, we show that models contextualize individual examples’ representations based on prior examples to adaptively reweight which demonstrations dominate the FV: attention shifts toward examples that are more informative and less ambiguous under the context. Finally, a causal decomposition separates Query–Key routing from Value updates, finding that contextualization’s most consistent contributions to FV quality arise from Query–Key alignment–particularly in ambiguous settings–while Value-mediated effects are more heterogeneous. Together, these results unify additive superposition with context-dependent attention reweighting into a mechanistic, testable account of how few-shot prompts implement tasks.
APA
Wang, E., Wang, Y., Bakalova, A. & Hahn, M.. (2026). How Few-Shot Examples Add Up: A Causal Decomposition of Function Vectors in In-Context Learning. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:130289-130358 Available from https://proceedings.mlr.press/v306/wang26hp.html.

Related Material