Logarithmic Width Suffices for Robust Memorization

Amitsour Egosi, Gilad Yehudai, Ohad Shamir
Proceedings of Thirty Eighth Conference on Learning Theory, PMLR 291:1638-1690, 2025.

Abstract

The memorization capacity of neural networks with a given architecture has been thoroughly studied in many works. Specifically, it is well-known that memorizing $N$ samples can be done using a network of constant width, independent of $N$. However, the required constructions are often quite delicate. In this paper, we consider the natural question of how well feedforward ReLU neural networks can memorize \emph{robustly}, namely while being able to withstand adversarial perturbations of a given radius. We establish both upper and lower bounds on the possible radius for general $l_p$ norms, implying (among other things) that width \emph{logarithmic} in the number of input samples is necessary and sufficient to achieve robust memorization (with robustness radius independent of $N$).

Cite this Paper

BibTeX
@InProceedings{pmlr-v291-egosi25a, title = {Logarithmic Width Suffices for Robust Memorization}, author = {Egosi, Amitsour and Yehudai, Gilad and Shamir, Ohad}, booktitle = {Proceedings of Thirty Eighth Conference on Learning Theory}, pages = {1638--1690}, year = {2025}, editor = {Haghtalab, Nika and Moitra, Ankur}, volume = {291}, series = {Proceedings of Machine Learning Research}, month = {30 Jun--04 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v291/main/assets/egosi25a/egosi25a.pdf}, url = {https://proceedings.mlr.press/v291/egosi25a.html}, abstract = { The memorization capacity of neural networks with a given architecture has been thoroughly studied in many works. Specifically, it is well-known that memorizing $N$ samples can be done using a network of constant width, independent of $N$. However, the required constructions are often quite delicate. In this paper, we consider the natural question of how well feedforward ReLU neural networks can memorize \emph{robustly}, namely while being able to withstand adversarial perturbations of a given radius. We establish both upper and lower bounds on the possible radius for general $l_p$ norms, implying (among other things) that width \emph{logarithmic} in the number of input samples is necessary and sufficient to achieve robust memorization (with robustness radius independent of $N$).} }
Endnote
%0 Conference Paper %T Logarithmic Width Suffices for Robust Memorization %A Amitsour Egosi %A Gilad Yehudai %A Ohad Shamir %B Proceedings of Thirty Eighth Conference on Learning Theory %C Proceedings of Machine Learning Research %D 2025 %E Nika Haghtalab %E Ankur Moitra %F pmlr-v291-egosi25a %I PMLR %P 1638--1690 %U https://proceedings.mlr.press/v291/egosi25a.html %V 291 %X The memorization capacity of neural networks with a given architecture has been thoroughly studied in many works. Specifically, it is well-known that memorizing $N$ samples can be done using a network of constant width, independent of $N$. However, the required constructions are often quite delicate. In this paper, we consider the natural question of how well feedforward ReLU neural networks can memorize \emph{robustly}, namely while being able to withstand adversarial perturbations of a given radius. We establish both upper and lower bounds on the possible radius for general $l_p$ norms, implying (among other things) that width \emph{logarithmic} in the number of input samples is necessary and sufficient to achieve robust memorization (with robustness radius independent of $N$).
APA
Egosi, A., Yehudai, G. & Shamir, O.. (2025). Logarithmic Width Suffices for Robust Memorization. Proceedings of Thirty Eighth Conference on Learning Theory, in Proceedings of Machine Learning Research 291:1638-1690 Available from https://proceedings.mlr.press/v291/egosi25a.html.

Related Material