Fisher8: Stabilizing Neural Heteroscedastic Regression via Output-Layer Fisher Geometry

Sumedh Vemuganti, Nickvash Kani
Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, PMLR 337:6885-6899, 2026.

Abstract

Training neural networks to jointly predict mean and uncertainty estimates from noisy observations can be unstable, prompting a series of independent stabilization efforts. We argue that these interventions highlight a common underlying issue where gradient steps are poorly aligned with the geometry of the loss landscape. To better align updates with local curvature, we derive Fisher8, an output-layer gradient correction that reorients and rescales updates using {Fisher} geometry rather than {Euclidean} geometry. Unlike past stabilizers, Fisher8 introduces no data-dependent hyperparameters beyond learning rate and admits an approximate KL trust radius between successive predictive distributions. We show that prior stabilizers converge on overlapping components of this geometric correction. Across multidimensional regression and representation-learning tasks, Fisher8 obtains superior likelihood–error tradeoffs, predicts calibrated uncertainty estimates, and learns rich uncertainty-aware feature spaces.

Cite this Paper

BibTeX
@InProceedings{pmlr-v337-vemuganti26a, title = {Fisher8: Stabilizing Neural Heteroscedastic Regression via Output-Layer {Fisher} Geometry}, author = {Vemuganti, Sumedh and Kani, Nickvash}, booktitle = {Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence}, pages = {6885--6899}, year = {2026}, editor = {Perković, Emilija and Malinsky, Daniel}, volume = {337}, series = {Proceedings of Machine Learning Research}, month = {17--21 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v337/main/assets/vemuganti26a/vemuganti26a.pdf}, url = {https://proceedings.mlr.press/v337/vemuganti26a.html}, abstract = {Training neural networks to jointly predict mean and uncertainty estimates from noisy observations can be unstable, prompting a series of independent stabilization efforts. We argue that these interventions highlight a common underlying issue where gradient steps are poorly aligned with the geometry of the loss landscape. To better align updates with local curvature, we derive Fisher8, an output-layer gradient correction that reorients and rescales updates using {Fisher} geometry rather than {Euclidean} geometry. Unlike past stabilizers, Fisher8 introduces no data-dependent hyperparameters beyond learning rate and admits an approximate KL trust radius between successive predictive distributions. We show that prior stabilizers converge on overlapping components of this geometric correction. Across multidimensional regression and representation-learning tasks, Fisher8 obtains superior likelihood–error tradeoffs, predicts calibrated uncertainty estimates, and learns rich uncertainty-aware feature spaces.} }
Endnote
%0 Conference Paper %T Fisher8: Stabilizing Neural Heteroscedastic Regression via Output-Layer Fisher Geometry %A Sumedh Vemuganti %A Nickvash Kani %B Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2026 %E Emilija Perković %E Daniel Malinsky %F pmlr-v337-vemuganti26a %I PMLR %P 6885--6899 %U https://proceedings.mlr.press/v337/vemuganti26a.html %V 337 %X Training neural networks to jointly predict mean and uncertainty estimates from noisy observations can be unstable, prompting a series of independent stabilization efforts. We argue that these interventions highlight a common underlying issue where gradient steps are poorly aligned with the geometry of the loss landscape. To better align updates with local curvature, we derive Fisher8, an output-layer gradient correction that reorients and rescales updates using {Fisher} geometry rather than {Euclidean} geometry. Unlike past stabilizers, Fisher8 introduces no data-dependent hyperparameters beyond learning rate and admits an approximate KL trust radius between successive predictive distributions. We show that prior stabilizers converge on overlapping components of this geometric correction. Across multidimensional regression and representation-learning tasks, Fisher8 obtains superior likelihood–error tradeoffs, predicts calibrated uncertainty estimates, and learns rich uncertainty-aware feature spaces.
APA
Vemuganti, S. & Kani, N.. (2026). Fisher8: Stabilizing Neural Heteroscedastic Regression via Output-Layer Fisher Geometry. Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 337:6885-6899 Available from https://proceedings.mlr.press/v337/vemuganti26a.html.

Related Material