InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition

Weidong Zhou, Fengze Liu, Binbin Liu, Ping Guo, Zijun Wang, Bingni Zhang, Yifan Zhang, Yifeng Yu, Xiaohuan Zhou, Taifeng Wang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:165450-165471, 2026.

Abstract

Upweighting high-quality data in LLM pretraining often improves performance, but in data-limited regimes, especially under overtraining, stronger upweighting increases repetition and can degrade performance. However, standard scaling laws do not reliably extrapolate across mixture recipes or under repetitions, making the selection for optimal data recipes at scaling underdetermined. To solve this, we introduce InfoLaw (Information Scaling Laws), a data-aware scaling framework that predicts loss from consumed tokens, model size, data mixture weights, and repetition. The key idea is to model pretraining as information accumulation, where quality controls information density and repetition induces scale-dependent diminishing returns. We first collect the model performance after training on datasets that vary in scale, quality distribution, and repetition level. Then we build up the modeling for information so that information accurately predicts those model performance. InfoLaw predicts performance on unseen data recipes and larger-scale runs (up to 7B, 425B tokens) with 0.15% mean and 0.96% max absolute error in loss, and it extrapolates reliably across overtraining levels, enabling efficient data-recipe selection under varying compute budgets.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-zhou26af, title = {{I}nfo{L}aw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition}, author = {Zhou, Weidong and Liu, Fengze and Liu, Binbin and Guo, Ping and Wang, Zijun and Zhang, Bingni and Zhang, Yifan and Yu, Yifeng and Zhou, Xiaohuan and Wang, Taifeng}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {165450--165471}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/zhou26af/zhou26af.pdf}, url = {https://proceedings.mlr.press/v306/zhou26af.html}, abstract = {Upweighting high-quality data in LLM pretraining often improves performance, but in data-limited regimes, especially under overtraining, stronger upweighting increases repetition and can degrade performance. However, standard scaling laws do not reliably extrapolate across mixture recipes or under repetitions, making the selection for optimal data recipes at scaling underdetermined. To solve this, we introduce InfoLaw (Information Scaling Laws), a data-aware scaling framework that predicts loss from consumed tokens, model size, data mixture weights, and repetition. The key idea is to model pretraining as information accumulation, where quality controls information density and repetition induces scale-dependent diminishing returns. We first collect the model performance after training on datasets that vary in scale, quality distribution, and repetition level. Then we build up the modeling for information so that information accurately predicts those model performance. InfoLaw predicts performance on unseen data recipes and larger-scale runs (up to 7B, 425B tokens) with 0.15% mean and 0.96% max absolute error in loss, and it extrapolates reliably across overtraining levels, enabling efficient data-recipe selection under varying compute budgets.} }
Endnote
%0 Conference Paper %T InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition %A Weidong Zhou %A Fengze Liu %A Binbin Liu %A Ping Guo %A Zijun Wang %A Bingni Zhang %A Yifan Zhang %A Yifeng Yu %A Xiaohuan Zhou %A Taifeng Wang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-zhou26af %I PMLR %P 165450--165471 %U https://proceedings.mlr.press/v306/zhou26af.html %V 306 %X Upweighting high-quality data in LLM pretraining often improves performance, but in data-limited regimes, especially under overtraining, stronger upweighting increases repetition and can degrade performance. However, standard scaling laws do not reliably extrapolate across mixture recipes or under repetitions, making the selection for optimal data recipes at scaling underdetermined. To solve this, we introduce InfoLaw (Information Scaling Laws), a data-aware scaling framework that predicts loss from consumed tokens, model size, data mixture weights, and repetition. The key idea is to model pretraining as information accumulation, where quality controls information density and repetition induces scale-dependent diminishing returns. We first collect the model performance after training on datasets that vary in scale, quality distribution, and repetition level. Then we build up the modeling for information so that information accurately predicts those model performance. InfoLaw predicts performance on unseen data recipes and larger-scale runs (up to 7B, 425B tokens) with 0.15% mean and 0.96% max absolute error in loss, and it extrapolates reliably across overtraining levels, enabling efficient data-recipe selection under varying compute budgets.
APA
Zhou, W., Liu, F., Liu, B., Guo, P., Wang, Z., Zhang, B., Zhang, Y., Yu, Y., Zhou, X. & Wang, T.. (2026). InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:165450-165471 Available from https://proceedings.mlr.press/v306/zhou26af.html.

Related Material