Manifold Metric: A Loss Landscape Approach for Predicting Model Performance

Pranshu Malviya, Jerry Huang, Aristide Baratin, Quentin Fournier, Sarath Chandar
Proceedings of The 4th Conference on Lifelong Learning Agents, PMLR 330:222-244, 2026.

Abstract

Determining the optimal model for a given task often requires training multiple models from scratch, which becomes impractical as dataset and model sizes grow. A more efficient alternative is to expand smaller pre-trained models, but this approach is underutilized due to a limited understanding of its impact on the training dynamics. Existing methods for quantifying this impact have notable limitations, including computation cost. To address this, we introduce a new perspective based on the loss landscape, which has been shown to contain a manifold of linearly connected minima. Specifically, we propose a metric that estimates the size of this manifold to study the impact of model expansion. Our experiments reveal a strong correlation between performance gains and our manifold metric, enabling more informed model comparison and offering a first step toward a geometry-driven approach for reliable model expansion. Notably, our metric outperforms other baselines, even when different types of expansion with equivalent number of parameters are applied to a model.

Cite this Paper

BibTeX
@InProceedings{pmlr-v330-malviya26a, title = {Manifold Metric: A Loss Landscape Approach for Predicting Model Performance}, author = {Malviya, Pranshu and Huang, Jerry and Baratin, Aristide and Fournier, Quentin and Chandar, Sarath}, booktitle = {Proceedings of The 4th Conference on Lifelong Learning Agents}, pages = {222--244}, year = {2026}, editor = {Chandar, Sarath and Pascanu, Razvan and Eaton, Eric and Liu, Bing and Mahmood, Rupam and Rannen-Triki, Amal}, volume = {330}, series = {Proceedings of Machine Learning Research}, month = {11--14 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v330/main/assets/malviya26a/malviya26a.pdf}, url = {https://proceedings.mlr.press/v330/malviya26a.html}, abstract = {Determining the optimal model for a given task often requires training multiple models from scratch, which becomes impractical as dataset and model sizes grow. A more efficient alternative is to expand smaller pre-trained models, but this approach is underutilized due to a limited understanding of its impact on the training dynamics. Existing methods for quantifying this impact have notable limitations, including computation cost. To address this, we introduce a new perspective based on the loss landscape, which has been shown to contain a manifold of linearly connected minima. Specifically, we propose a metric that estimates the size of this manifold to study the impact of model expansion. Our experiments reveal a strong correlation between performance gains and our manifold metric, enabling more informed model comparison and offering a first step toward a geometry-driven approach for reliable model expansion. Notably, our metric outperforms other baselines, even when different types of expansion with equivalent number of parameters are applied to a model.} }
Endnote
%0 Conference Paper %T Manifold Metric: A Loss Landscape Approach for Predicting Model Performance %A Pranshu Malviya %A Jerry Huang %A Aristide Baratin %A Quentin Fournier %A Sarath Chandar %B Proceedings of The 4th Conference on Lifelong Learning Agents %C Proceedings of Machine Learning Research %D 2026 %E Sarath Chandar %E Razvan Pascanu %E Eric Eaton %E Bing Liu %E Rupam Mahmood %E Amal Rannen-Triki %F pmlr-v330-malviya26a %I PMLR %P 222--244 %U https://proceedings.mlr.press/v330/malviya26a.html %V 330 %X Determining the optimal model for a given task often requires training multiple models from scratch, which becomes impractical as dataset and model sizes grow. A more efficient alternative is to expand smaller pre-trained models, but this approach is underutilized due to a limited understanding of its impact on the training dynamics. Existing methods for quantifying this impact have notable limitations, including computation cost. To address this, we introduce a new perspective based on the loss landscape, which has been shown to contain a manifold of linearly connected minima. Specifically, we propose a metric that estimates the size of this manifold to study the impact of model expansion. Our experiments reveal a strong correlation between performance gains and our manifold metric, enabling more informed model comparison and offering a first step toward a geometry-driven approach for reliable model expansion. Notably, our metric outperforms other baselines, even when different types of expansion with equivalent number of parameters are applied to a model.
APA
Malviya, P., Huang, J., Baratin, A., Fournier, Q. & Chandar, S.. (2026). Manifold Metric: A Loss Landscape Approach for Predicting Model Performance. Proceedings of The 4th Conference on Lifelong Learning Agents, in Proceedings of Machine Learning Research 330:222-244 Available from https://proceedings.mlr.press/v330/malviya26a.html.

Related Material