Understanding Nonlinear Implicit Bias via Region Counts in Input Space

Jingwei Li, Jing Xu, Zifan Wang, Huishuai Zhang, Jingzhao Zhang
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:36597-36613, 2025.

Abstract

One explanation for the strong generalization ability of neural networks is implicit bias. Yet, the definition and mechanism of implicit bias in non-linear contexts remains little understood. In this work, we propose to characterize implicit bias by the count of connected regions in the input space with the same predicted label. Compared with parameter-dependent metrics (e.g., norm or normalized margin), region count can be better adapted to nonlinear, overparameterized models, because it is determined by the function mapping and is invariant to reparametrization. Empirically, we found that small region counts align with geometrically simple decision boundaries and correlate well with good generalization performance. We also observe that good hyper-parameter choices such as larger learning rates and smaller batch sizes can induce small region counts. We further establish the theoretical connections and explain how larger learning rate can induce small region counts in neural networks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-li25di, title = {Understanding Nonlinear Implicit Bias via Region Counts in Input Space}, author = {Li, Jingwei and Xu, Jing and Wang, Zifan and Zhang, Huishuai and Zhang, Jingzhao}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {36597--36613}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/li25di/li25di.pdf}, url = {https://proceedings.mlr.press/v267/li25di.html}, abstract = {One explanation for the strong generalization ability of neural networks is implicit bias. Yet, the definition and mechanism of implicit bias in non-linear contexts remains little understood. In this work, we propose to characterize implicit bias by the count of connected regions in the input space with the same predicted label. Compared with parameter-dependent metrics (e.g., norm or normalized margin), region count can be better adapted to nonlinear, overparameterized models, because it is determined by the function mapping and is invariant to reparametrization. Empirically, we found that small region counts align with geometrically simple decision boundaries and correlate well with good generalization performance. We also observe that good hyper-parameter choices such as larger learning rates and smaller batch sizes can induce small region counts. We further establish the theoretical connections and explain how larger learning rate can induce small region counts in neural networks.} }
Endnote
%0 Conference Paper %T Understanding Nonlinear Implicit Bias via Region Counts in Input Space %A Jingwei Li %A Jing Xu %A Zifan Wang %A Huishuai Zhang %A Jingzhao Zhang %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-li25di %I PMLR %P 36597--36613 %U https://proceedings.mlr.press/v267/li25di.html %V 267 %X One explanation for the strong generalization ability of neural networks is implicit bias. Yet, the definition and mechanism of implicit bias in non-linear contexts remains little understood. In this work, we propose to characterize implicit bias by the count of connected regions in the input space with the same predicted label. Compared with parameter-dependent metrics (e.g., norm or normalized margin), region count can be better adapted to nonlinear, overparameterized models, because it is determined by the function mapping and is invariant to reparametrization. Empirically, we found that small region counts align with geometrically simple decision boundaries and correlate well with good generalization performance. We also observe that good hyper-parameter choices such as larger learning rates and smaller batch sizes can induce small region counts. We further establish the theoretical connections and explain how larger learning rate can induce small region counts in neural networks.
APA
Li, J., Xu, J., Wang, Z., Zhang, H. & Zhang, J.. (2025). Understanding Nonlinear Implicit Bias via Region Counts in Input Space. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:36597-36613 Available from https://proceedings.mlr.press/v267/li25di.html.

Related Material