Optimizing Rank for High-Fidelity Implicit Neural Representations

Julian Mcginnis, Florian A. Hölzl, Suprosanna Shit, Florentin Bieder, Paul Friedrich, Mark Mühlau, Bjoern Menze, Daniel Rueckert, Benedikt Wiestler
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:87267-87296, 2026.

Abstract

Implicit Neural Representations (INRs) based on vanilla Multi-Layer Perceptrons (MLPs) are widely believed to be incapable of representing high-frequency content. This has directed research efforts towards architectural interventions, such as coordinate embeddings or specialized activation functions, to represent high-frequency signals. In this paper, we challenge the notion that the low-frequency bias of vanilla MLPs is an intrinsic, architectural limitation to learn high-frequency content, but instead a symptom of stable rank degradation during training. We empirically demonstrate that regulating the network’s rank during training substantially improves the fidelity of the learned signal, rendering even simple MLP architectures expressive. Extensive experiments show that using optimizers like Muon, with high-rank, near-orthogonal updates, consistently enhances INR architectures even beyond simple ReLU MLPs. These substantial improvements hold across a diverse range of domains, including natural and medical images and novel view synthesis, with up to +9 dB PSNR over the samearchitecture. Code is available here.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-mcginnis26a, title = {Optimizing Rank for High-Fidelity Implicit Neural Representations}, author = {Mcginnis, Julian and H\"{o}lzl, Florian A. and Shit, Suprosanna and Bieder, Florentin and Friedrich, Paul and M\"{u}hlau, Mark and Menze, Bjoern and Rueckert, Daniel and Wiestler, Benedikt}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {87267--87296}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/mcginnis26a/mcginnis26a.pdf}, url = {https://proceedings.mlr.press/v306/mcginnis26a.html}, abstract = {Implicit Neural Representations (INRs) based on vanilla Multi-Layer Perceptrons (MLPs) are widely believed to be incapable of representing high-frequency content. This has directed research efforts towards architectural interventions, such as coordinate embeddings or specialized activation functions, to represent high-frequency signals. In this paper, we challenge the notion that the low-frequency bias of vanilla MLPs is an intrinsic, architectural limitation to learn high-frequency content, but instead a symptom of stable rank degradation during training. We empirically demonstrate that regulating the network’s rank during training substantially improves the fidelity of the learned signal, rendering even simple MLP architectures expressive. Extensive experiments show that using optimizers like Muon, with high-rank, near-orthogonal updates, consistently enhances INR architectures even beyond simple ReLU MLPs. These substantial improvements hold across a diverse range of domains, including natural and medical images and novel view synthesis, with up to +9 dB PSNR over the samearchitecture. Code is available here.} }
Endnote
%0 Conference Paper %T Optimizing Rank for High-Fidelity Implicit Neural Representations %A Julian Mcginnis %A Florian A. Hölzl %A Suprosanna Shit %A Florentin Bieder %A Paul Friedrich %A Mark Mühlau %A Bjoern Menze %A Daniel Rueckert %A Benedikt Wiestler %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-mcginnis26a %I PMLR %P 87267--87296 %U https://proceedings.mlr.press/v306/mcginnis26a.html %V 306 %X Implicit Neural Representations (INRs) based on vanilla Multi-Layer Perceptrons (MLPs) are widely believed to be incapable of representing high-frequency content. This has directed research efforts towards architectural interventions, such as coordinate embeddings or specialized activation functions, to represent high-frequency signals. In this paper, we challenge the notion that the low-frequency bias of vanilla MLPs is an intrinsic, architectural limitation to learn high-frequency content, but instead a symptom of stable rank degradation during training. We empirically demonstrate that regulating the network’s rank during training substantially improves the fidelity of the learned signal, rendering even simple MLP architectures expressive. Extensive experiments show that using optimizers like Muon, with high-rank, near-orthogonal updates, consistently enhances INR architectures even beyond simple ReLU MLPs. These substantial improvements hold across a diverse range of domains, including natural and medical images and novel view synthesis, with up to +9 dB PSNR over the samearchitecture. Code is available here.
APA
Mcginnis, J., Hölzl, F.A., Shit, S., Bieder, F., Friedrich, P., Mühlau, M., Menze, B., Rueckert, D. & Wiestler, B.. (2026). Optimizing Rank for High-Fidelity Implicit Neural Representations. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:87267-87296 Available from https://proceedings.mlr.press/v306/mcginnis26a.html.

Related Material