Inverse Depth Scaling From Most Layers Being Similar

Yizhou Liu, Sara Kangaslahti, Ziming Liu, Jeff Gore
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:77093-77118, 2026.

Abstract

Neural scaling laws relate loss to model size in large language models (LLMs), yet depth and width may contribute to performance differently, requiring more detailed studies. Here, we quantify how depth affects loss via analysis of LLMs and toy residual networks. We find loss scales inversely proportional to depth in LLMs, probably due to functionally similar layers reducing error through ensemble averaging rather than compositional learning or discretizing smooth dynamics. This regime is inefficient yet robust and may arise from the architectural bias of residual networks and target functions incompatible with smooth dynamics. The findings suggest that improving LLM efficiency may require architectural innovations to encourage compositional use of depth.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-liu26bt, title = {Inverse Depth Scaling From Most Layers Being Similar}, author = {Liu, Yizhou and Kangaslahti, Sara and Liu, Ziming and Gore, Jeff}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {77093--77118}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/liu26bt/liu26bt.pdf}, url = {https://proceedings.mlr.press/v306/liu26bt.html}, abstract = {Neural scaling laws relate loss to model size in large language models (LLMs), yet depth and width may contribute to performance differently, requiring more detailed studies. Here, we quantify how depth affects loss via analysis of LLMs and toy residual networks. We find loss scales inversely proportional to depth in LLMs, probably due to functionally similar layers reducing error through ensemble averaging rather than compositional learning or discretizing smooth dynamics. This regime is inefficient yet robust and may arise from the architectural bias of residual networks and target functions incompatible with smooth dynamics. The findings suggest that improving LLM efficiency may require architectural innovations to encourage compositional use of depth.} }
Endnote
%0 Conference Paper %T Inverse Depth Scaling From Most Layers Being Similar %A Yizhou Liu %A Sara Kangaslahti %A Ziming Liu %A Jeff Gore %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-liu26bt %I PMLR %P 77093--77118 %U https://proceedings.mlr.press/v306/liu26bt.html %V 306 %X Neural scaling laws relate loss to model size in large language models (LLMs), yet depth and width may contribute to performance differently, requiring more detailed studies. Here, we quantify how depth affects loss via analysis of LLMs and toy residual networks. We find loss scales inversely proportional to depth in LLMs, probably due to functionally similar layers reducing error through ensemble averaging rather than compositional learning or discretizing smooth dynamics. This regime is inefficient yet robust and may arise from the architectural bias of residual networks and target functions incompatible with smooth dynamics. The findings suggest that improving LLM efficiency may require architectural innovations to encourage compositional use of depth.
APA
Liu, Y., Kangaslahti, S., Liu, Z. & Gore, J.. (2026). Inverse Depth Scaling From Most Layers Being Similar. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:77093-77118 Available from https://proceedings.mlr.press/v306/liu26bt.html.

Related Material