Why ReLU? A Bit-Model Dichotomy for Deep Network Training

Ilan Doron-Arad, Elchanan Mossel
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:26202-26226, 2026.

Abstract

Theoretical analyses of Empirical Risk Minimization (ERM) are standardly framed within the Real-RAM model of computation. In this setting, training even simple neural networks is known to be $\exists \mathbb{R}$-complete - a complexity class believed to be harder than NP, characterizing the difficulty of solving systems of polynomial inequalities over the real numbers. However, this algebraic framework diverges from the reality of digital computation with finite-precision hardware. In this work, we analyze the theoretical complexity of ERM under a realistic bit-level model (ERM-bit), where network parameters and inputs are constrained to be rational numbers with polynomially bounded bit-lengths. Under this model, we reveal a sharp dichotomy in tractability governed by the activation function: for deep networks with any polynomial activation with rational coefficients and degree at least $2$, deciding ERM-bit is #P-hard, determining the sign of a single partial derivative is unlikely to be in BPP, and deciding a specific bit in the gradient is #P-hard. In contrast, for piecewise-linear activations such as ReLU, precision requirements remain manageable: ERM-bit is in NP, indeed NP-complete, and standard backpropagation runs in polynomial time, showing that finite-precision constraints are not merely implementation details but fundamental determinants of learnability.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-doron-arad26a, title = {Why {R}e{LU}? {A} Bit-Model Dichotomy for Deep Network Training}, author = {Doron-Arad, Ilan and Mossel, Elchanan}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {26202--26226}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/doron-arad26a/doron-arad26a.pdf}, url = {https://proceedings.mlr.press/v306/doron-arad26a.html}, abstract = {Theoretical analyses of Empirical Risk Minimization (ERM) are standardly framed within the Real-RAM model of computation. In this setting, training even simple neural networks is known to be $\exists \mathbb{R}$-complete - a complexity class believed to be harder than NP, characterizing the difficulty of solving systems of polynomial inequalities over the real numbers. However, this algebraic framework diverges from the reality of digital computation with finite-precision hardware. In this work, we analyze the theoretical complexity of ERM under a realistic bit-level model (ERM-bit), where network parameters and inputs are constrained to be rational numbers with polynomially bounded bit-lengths. Under this model, we reveal a sharp dichotomy in tractability governed by the activation function: for deep networks with any polynomial activation with rational coefficients and degree at least $2$, deciding ERM-bit is #P-hard, determining the sign of a single partial derivative is unlikely to be in BPP, and deciding a specific bit in the gradient is #P-hard. In contrast, for piecewise-linear activations such as ReLU, precision requirements remain manageable: ERM-bit is in NP, indeed NP-complete, and standard backpropagation runs in polynomial time, showing that finite-precision constraints are not merely implementation details but fundamental determinants of learnability.} }
Endnote
%0 Conference Paper %T Why ReLU? A Bit-Model Dichotomy for Deep Network Training %A Ilan Doron-Arad %A Elchanan Mossel %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-doron-arad26a %I PMLR %P 26202--26226 %U https://proceedings.mlr.press/v306/doron-arad26a.html %V 306 %X Theoretical analyses of Empirical Risk Minimization (ERM) are standardly framed within the Real-RAM model of computation. In this setting, training even simple neural networks is known to be $\exists \mathbb{R}$-complete - a complexity class believed to be harder than NP, characterizing the difficulty of solving systems of polynomial inequalities over the real numbers. However, this algebraic framework diverges from the reality of digital computation with finite-precision hardware. In this work, we analyze the theoretical complexity of ERM under a realistic bit-level model (ERM-bit), where network parameters and inputs are constrained to be rational numbers with polynomially bounded bit-lengths. Under this model, we reveal a sharp dichotomy in tractability governed by the activation function: for deep networks with any polynomial activation with rational coefficients and degree at least $2$, deciding ERM-bit is #P-hard, determining the sign of a single partial derivative is unlikely to be in BPP, and deciding a specific bit in the gradient is #P-hard. In contrast, for piecewise-linear activations such as ReLU, precision requirements remain manageable: ERM-bit is in NP, indeed NP-complete, and standard backpropagation runs in polynomial time, showing that finite-precision constraints are not merely implementation details but fundamental determinants of learnability.
APA
Doron-Arad, I. & Mossel, E.. (2026). Why ReLU? A Bit-Model Dichotomy for Deep Network Training. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:26202-26226 Available from https://proceedings.mlr.press/v306/doron-arad26a.html.

Related Material