Active Budget Allocation for Efficient Scaling Law Estimation via Surrogate-Guided Pruning

Viktoria Schram, Markus Hiller, Daniel Beck, Trevor Cohn
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:108421-108453, 2026.

Abstract

Predicting model performance at larger scales enables the design of training strategies and architectures tailored to specific performance targets. Empirical scaling law research identifies functional forms to aid this prediction task. These describe the relationship between loss and compute using a loss-compute frontier defined by learning curves. Due to the empirical nature of this approach, the computational burden is substantial, making strategic resource allocation essential – yet it remains surprisingly underexplored. In this work, we address this shortcoming by exploring the suitability of Successive Halving (SH) and SH combined with parametric and non-parametric surrogate models. In addition to enabling a more systematic allocation of a given compute budget, our findings show that SH paired with surrogate models yields a set of learning curves that includes one with a lower loss-compute value than what naive uniform allocation or an SH-only approach can obtain. Our experiments demonstrate mean relative improvements of up to $2.84%$ and $5.47%$ on real-world and synthetic learning curve datasets. This strategic resource allocation enables us to obtain accurate scaling laws at significantly reduced computational costs, saving up to $98.7%$ over the traditional exhaustive approach.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-schram26a, title = {Active Budget Allocation for Efficient Scaling Law Estimation via Surrogate-Guided Pruning}, author = {Schram, Viktoria and Hiller, Markus and Beck, Daniel and Cohn, Trevor}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {108421--108453}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/schram26a/schram26a.pdf}, url = {https://proceedings.mlr.press/v306/schram26a.html}, abstract = {Predicting model performance at larger scales enables the design of training strategies and architectures tailored to specific performance targets. Empirical scaling law research identifies functional forms to aid this prediction task. These describe the relationship between loss and compute using a loss-compute frontier defined by learning curves. Due to the empirical nature of this approach, the computational burden is substantial, making strategic resource allocation essential – yet it remains surprisingly underexplored. In this work, we address this shortcoming by exploring the suitability of Successive Halving (SH) and SH combined with parametric and non-parametric surrogate models. In addition to enabling a more systematic allocation of a given compute budget, our findings show that SH paired with surrogate models yields a set of learning curves that includes one with a lower loss-compute value than what naive uniform allocation or an SH-only approach can obtain. Our experiments demonstrate mean relative improvements of up to $2.84%$ and $5.47%$ on real-world and synthetic learning curve datasets. This strategic resource allocation enables us to obtain accurate scaling laws at significantly reduced computational costs, saving up to $98.7%$ over the traditional exhaustive approach.} }
Endnote
%0 Conference Paper %T Active Budget Allocation for Efficient Scaling Law Estimation via Surrogate-Guided Pruning %A Viktoria Schram %A Markus Hiller %A Daniel Beck %A Trevor Cohn %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-schram26a %I PMLR %P 108421--108453 %U https://proceedings.mlr.press/v306/schram26a.html %V 306 %X Predicting model performance at larger scales enables the design of training strategies and architectures tailored to specific performance targets. Empirical scaling law research identifies functional forms to aid this prediction task. These describe the relationship between loss and compute using a loss-compute frontier defined by learning curves. Due to the empirical nature of this approach, the computational burden is substantial, making strategic resource allocation essential – yet it remains surprisingly underexplored. In this work, we address this shortcoming by exploring the suitability of Successive Halving (SH) and SH combined with parametric and non-parametric surrogate models. In addition to enabling a more systematic allocation of a given compute budget, our findings show that SH paired with surrogate models yields a set of learning curves that includes one with a lower loss-compute value than what naive uniform allocation or an SH-only approach can obtain. Our experiments demonstrate mean relative improvements of up to $2.84%$ and $5.47%$ on real-world and synthetic learning curve datasets. This strategic resource allocation enables us to obtain accurate scaling laws at significantly reduced computational costs, saving up to $98.7%$ over the traditional exhaustive approach.
APA
Schram, V., Hiller, M., Beck, D. & Cohn, T.. (2026). Active Budget Allocation for Efficient Scaling Law Estimation via Surrogate-Guided Pruning. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:108421-108453 Available from https://proceedings.mlr.press/v306/schram26a.html.

Related Material