The Local Learning Coefficient: A Singularity-Aware Complexity Measure

Edmund Lau, Zach Furman, George Wang, Daniel Murfet, Susan Wei
Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, PMLR 258:244-252, 2025.

Abstract

The Local Learning Coefficient (LLC) is introduced as a novel complexity measure for deep neural networks (DNNs). Recognizing the limitations of traditional complexity measures, the LLC leverages Singular Learning Theory (SLT), which has long recognized the significance of singularities in the loss landscape geometry. This paper provides an extensive exploration of the LLC’s theoretical underpinnings, offering both a clear definition and intuitive insights into its application. Moreover, we propose a new scalable estimator for the LLC, which is then effectively applied across diverse architectures including deep linear networks up to 100M parameters, ResNet image models, and transformer language models. Empirical evidence suggests that the LLC provides valuable insights into how training heuristics might influence the effective complexity of DNNs. Ultimately, the LLC emerges as a crucial tool for reconciling the apparent contradiction between deep learning’s complexity and the principle of parsimony.

Cite this Paper

BibTeX
@InProceedings{pmlr-v258-lau25a, title = {The Local Learning Coefficient: A Singularity-Aware Complexity Measure}, author = {Lau, Edmund and Furman, Zach and Wang, George and Murfet, Daniel and Wei, Susan}, booktitle = {Proceedings of The 28th International Conference on Artificial Intelligence and Statistics}, pages = {244--252}, year = {2025}, editor = {Li, Yingzhen and Mandt, Stephan and Agrawal, Shipra and Khan, Emtiyaz}, volume = {258}, series = {Proceedings of Machine Learning Research}, month = {03--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v258/main/assets/lau25a/lau25a.pdf}, url = {https://proceedings.mlr.press/v258/lau25a.html}, abstract = {The Local Learning Coefficient (LLC) is introduced as a novel complexity measure for deep neural networks (DNNs). Recognizing the limitations of traditional complexity measures, the LLC leverages Singular Learning Theory (SLT), which has long recognized the significance of singularities in the loss landscape geometry. This paper provides an extensive exploration of the LLC’s theoretical underpinnings, offering both a clear definition and intuitive insights into its application. Moreover, we propose a new scalable estimator for the LLC, which is then effectively applied across diverse architectures including deep linear networks up to 100M parameters, ResNet image models, and transformer language models. Empirical evidence suggests that the LLC provides valuable insights into how training heuristics might influence the effective complexity of DNNs. Ultimately, the LLC emerges as a crucial tool for reconciling the apparent contradiction between deep learning’s complexity and the principle of parsimony.} }
Endnote
%0 Conference Paper %T The Local Learning Coefficient: A Singularity-Aware Complexity Measure %A Edmund Lau %A Zach Furman %A George Wang %A Daniel Murfet %A Susan Wei %B Proceedings of The 28th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2025 %E Yingzhen Li %E Stephan Mandt %E Shipra Agrawal %E Emtiyaz Khan %F pmlr-v258-lau25a %I PMLR %P 244--252 %U https://proceedings.mlr.press/v258/lau25a.html %V 258 %X The Local Learning Coefficient (LLC) is introduced as a novel complexity measure for deep neural networks (DNNs). Recognizing the limitations of traditional complexity measures, the LLC leverages Singular Learning Theory (SLT), which has long recognized the significance of singularities in the loss landscape geometry. This paper provides an extensive exploration of the LLC’s theoretical underpinnings, offering both a clear definition and intuitive insights into its application. Moreover, we propose a new scalable estimator for the LLC, which is then effectively applied across diverse architectures including deep linear networks up to 100M parameters, ResNet image models, and transformer language models. Empirical evidence suggests that the LLC provides valuable insights into how training heuristics might influence the effective complexity of DNNs. Ultimately, the LLC emerges as a crucial tool for reconciling the apparent contradiction between deep learning’s complexity and the principle of parsimony.
APA
Lau, E., Furman, Z., Wang, G., Murfet, D. & Wei, S.. (2025). The Local Learning Coefficient: A Singularity-Aware Complexity Measure. Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 258:244-252 Available from https://proceedings.mlr.press/v258/lau25a.html.

Related Material