Causally Evaluating the Learnability of Formal Language Tasks

Vésteinn Snæbjarnarson, Anej Svete, Josef Valvoda, Reda Boumasmoud, Brian Dusell, Ryan Cotterell
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:114024-114050, 2026.

Abstract

Language models, as multi-task learners, acquire a wide range of abilities during training. A fundamental question is how much task-specific data is needed to learn a given task. Answering this for natural language is difficult: tasks are hard to delineate and can confound one another. To rigorously investigate the relationship between data frequency and learnability, we turn to a controlled setting using formal languages induced from probabilistic finite automata. These serve as a methodological testbed to demonstrate that standard correlational evaluation practices are inherently flawed. To enable causal analysis, we introduce the binning semiring, an algebraic object that lets us control how often a targeted property occurs in a sampled corpus. We formulate the experimental pipeline as a causal graphical model and derive decomposed Kullback–Leibler divergence metrics to measure the learnability of specific sub-tasks. Our experiments show that evaluating learnability without causal intervention leads to incorrect conclusions due to confounders in correlational analysis, and serve as a warning about correlational pitfalls in natural-language settings.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-snaebjarnarson26a, title = {Causally Evaluating the Learnability of Formal Language Tasks}, author = {Sn{\ae}bjarnarson, V\'{e}steinn and Svete, Anej and Valvoda, Josef and Boumasmoud, Reda and Dusell, Brian and Cotterell, Ryan}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {114024--114050}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/snaebjarnarson26a/snaebjarnarson26a.pdf}, url = {https://proceedings.mlr.press/v306/snaebjarnarson26a.html}, abstract = {Language models, as multi-task learners, acquire a wide range of abilities during training. A fundamental question is how much task-specific data is needed to learn a given task. Answering this for natural language is difficult: tasks are hard to delineate and can confound one another. To rigorously investigate the relationship between data frequency and learnability, we turn to a controlled setting using formal languages induced from probabilistic finite automata. These serve as a methodological testbed to demonstrate that standard correlational evaluation practices are inherently flawed. To enable causal analysis, we introduce the binning semiring, an algebraic object that lets us control how often a targeted property occurs in a sampled corpus. We formulate the experimental pipeline as a causal graphical model and derive decomposed Kullback–Leibler divergence metrics to measure the learnability of specific sub-tasks. Our experiments show that evaluating learnability without causal intervention leads to incorrect conclusions due to confounders in correlational analysis, and serve as a warning about correlational pitfalls in natural-language settings.} }
Endnote
%0 Conference Paper %T Causally Evaluating the Learnability of Formal Language Tasks %A Vésteinn Snæbjarnarson %A Anej Svete %A Josef Valvoda %A Reda Boumasmoud %A Brian Dusell %A Ryan Cotterell %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-snaebjarnarson26a %I PMLR %P 114024--114050 %U https://proceedings.mlr.press/v306/snaebjarnarson26a.html %V 306 %X Language models, as multi-task learners, acquire a wide range of abilities during training. A fundamental question is how much task-specific data is needed to learn a given task. Answering this for natural language is difficult: tasks are hard to delineate and can confound one another. To rigorously investigate the relationship between data frequency and learnability, we turn to a controlled setting using formal languages induced from probabilistic finite automata. These serve as a methodological testbed to demonstrate that standard correlational evaluation practices are inherently flawed. To enable causal analysis, we introduce the binning semiring, an algebraic object that lets us control how often a targeted property occurs in a sampled corpus. We formulate the experimental pipeline as a causal graphical model and derive decomposed Kullback–Leibler divergence metrics to measure the learnability of specific sub-tasks. Our experiments show that evaluating learnability without causal intervention leads to incorrect conclusions due to confounders in correlational analysis, and serve as a warning about correlational pitfalls in natural-language settings.
APA
Snæbjarnarson, V., Svete, A., Valvoda, J., Boumasmoud, R., Dusell, B. & Cotterell, R.. (2026). Causally Evaluating the Learnability of Formal Language Tasks. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:114024-114050 Available from https://proceedings.mlr.press/v306/snaebjarnarson26a.html.

Related Material