Implicit meta-learning may lead language models to trust more reliable sources

Dmitrii Krasheninnikov, Egor Krasheninnikov, Bruno Kacper Mlodozeniec, Tegan Maharaj, David Krueger
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:25534-25559, 2024.

Abstract

We demonstrate that large language models (LLMs) may learn indicators of document usefulness and modulate their updates accordingly. We introduce random strings ("tags") as indicators of usefulness in a synthetic fine-tuning dataset. Fine-tuning on this dataset leads to implicit meta-learning (IML): in further fine-tuning, the model updates to make more use of text that is tagged as useful. We perform a thorough empirical investigation of this phenomenon, finding (among other things) that (i) it occurs in both pretrained LLMs and those trained from scratch, as well as on a vision task, and (ii) larger models and smaller batch sizes tend to give more IML. We also use probing to examine how IML changes the way models store knowledge in their parameters. Finally, we reflect on what our results might imply about the capabilities, risks, and controllability of future AI systems.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-krasheninnikov24a, title = {Implicit meta-learning may lead language models to trust more reliable sources}, author = {Krasheninnikov, Dmitrii and Krasheninnikov, Egor and Mlodozeniec, Bruno Kacper and Maharaj, Tegan and Krueger, David}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {25534--25559}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/krasheninnikov24a/krasheninnikov24a.pdf}, url = {https://proceedings.mlr.press/v235/krasheninnikov24a.html}, abstract = {We demonstrate that large language models (LLMs) may learn indicators of document usefulness and modulate their updates accordingly. We introduce random strings ("tags") as indicators of usefulness in a synthetic fine-tuning dataset. Fine-tuning on this dataset leads to implicit meta-learning (IML): in further fine-tuning, the model updates to make more use of text that is tagged as useful. We perform a thorough empirical investigation of this phenomenon, finding (among other things) that (i) it occurs in both pretrained LLMs and those trained from scratch, as well as on a vision task, and (ii) larger models and smaller batch sizes tend to give more IML. We also use probing to examine how IML changes the way models store knowledge in their parameters. Finally, we reflect on what our results might imply about the capabilities, risks, and controllability of future AI systems.} }
Endnote
%0 Conference Paper %T Implicit meta-learning may lead language models to trust more reliable sources %A Dmitrii Krasheninnikov %A Egor Krasheninnikov %A Bruno Kacper Mlodozeniec %A Tegan Maharaj %A David Krueger %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-krasheninnikov24a %I PMLR %P 25534--25559 %U https://proceedings.mlr.press/v235/krasheninnikov24a.html %V 235 %X We demonstrate that large language models (LLMs) may learn indicators of document usefulness and modulate their updates accordingly. We introduce random strings ("tags") as indicators of usefulness in a synthetic fine-tuning dataset. Fine-tuning on this dataset leads to implicit meta-learning (IML): in further fine-tuning, the model updates to make more use of text that is tagged as useful. We perform a thorough empirical investigation of this phenomenon, finding (among other things) that (i) it occurs in both pretrained LLMs and those trained from scratch, as well as on a vision task, and (ii) larger models and smaller batch sizes tend to give more IML. We also use probing to examine how IML changes the way models store knowledge in their parameters. Finally, we reflect on what our results might imply about the capabilities, risks, and controllability of future AI systems.
APA
Krasheninnikov, D., Krasheninnikov, E., Mlodozeniec, B.K., Maharaj, T. & Krueger, D.. (2024). Implicit meta-learning may lead language models to trust more reliable sources. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:25534-25559 Available from https://proceedings.mlr.press/v235/krasheninnikov24a.html.

Related Material