Structured Covariance Modeling Using Learned Mixture-of-Bases for Uncertainty in 3D Segmentation

Peter J.T. Kampen, Andreas With Aspe, Kristine Aavild Juhl, Anders Nymark Christensen, Morten Rieger Hannemose, Anders Dahl, Rasmus Reinhold Paulsen, Josefine Vilsbøll Sundgaard
Proceedings of the 7th Northern Lights Deep Learning Conference (NLDL), PMLR 307:191-200, 2026.

Abstract

Accurate segmentation is essential in error-critical domains such as medical imaging, where outputs support clinical decisions. Probabilistic models like the Stochastic Segmentation Network (SSN) enable uncertainty quantification, but existing methods typically use low-rank plus diagonal covariance structures that struggle to capture both global and local spatial correlations, limiting performance gains over deterministic models. We revisit low-rank formulations and introduce two approaches - Single-Basis and Mixture-of-Bases decompositions - that project predicted noise onto learned covariance bases, either globally or within partitioned volume blocks. This yields richer, more flexible uncertainty modeling with minimal parameter overhead. On the most challenging organs in the 3D TotalSegmentator CT dataset, our methods significantly improve Dice scores over deterministic and baseline stochastic models while preserving strong calibration, with the Mixture-of-Bases performing best. These findings show that basis-driven covariance modeling can enhance segmentation accuracy and uncertainty estimation in 3D medical imaging.

Cite this Paper

BibTeX
@InProceedings{pmlr-v307-kampen26a, title = {Structured Covariance Modeling Using Learned Mixture-of-Bases for Uncertainty in 3D Segmentation}, author = {Kampen, Peter J.T. and Aspe, Andreas With and Juhl, Kristine Aavild and Christensen, Anders Nymark and Hannemose, Morten Rieger and Dahl, Anders and Paulsen, Rasmus Reinhold and Sundgaard, Josefine Vilsb{\o}ll}, booktitle = {Proceedings of the 7th Northern Lights Deep Learning Conference (NLDL)}, pages = {191--200}, year = {2026}, editor = {Kim, Hyeongji and Ramírez Rivera, Adín and Ricaud, Benjamin}, volume = {307}, series = {Proceedings of Machine Learning Research}, month = {06--08 Jan}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v307/main/assets/kampen26a/kampen26a.pdf}, url = {https://proceedings.mlr.press/v307/kampen26a.html}, abstract = {Accurate segmentation is essential in error-critical domains such as medical imaging, where outputs support clinical decisions. Probabilistic models like the Stochastic Segmentation Network (SSN) enable uncertainty quantification, but existing methods typically use low-rank plus diagonal covariance structures that struggle to capture both global and local spatial correlations, limiting performance gains over deterministic models. We revisit low-rank formulations and introduce two approaches - Single-Basis and Mixture-of-Bases decompositions - that project predicted noise onto learned covariance bases, either globally or within partitioned volume blocks. This yields richer, more flexible uncertainty modeling with minimal parameter overhead. On the most challenging organs in the 3D TotalSegmentator CT dataset, our methods significantly improve Dice scores over deterministic and baseline stochastic models while preserving strong calibration, with the Mixture-of-Bases performing best. These findings show that basis-driven covariance modeling can enhance segmentation accuracy and uncertainty estimation in 3D medical imaging.} }
Endnote
%0 Conference Paper %T Structured Covariance Modeling Using Learned Mixture-of-Bases for Uncertainty in 3D Segmentation %A Peter J.T. Kampen %A Andreas With Aspe %A Kristine Aavild Juhl %A Anders Nymark Christensen %A Morten Rieger Hannemose %A Anders Dahl %A Rasmus Reinhold Paulsen %A Josefine Vilsbøll Sundgaard %B Proceedings of the 7th Northern Lights Deep Learning Conference (NLDL) %C Proceedings of Machine Learning Research %D 2026 %E Hyeongji Kim %E Adín Ramírez Rivera %E Benjamin Ricaud %F pmlr-v307-kampen26a %I PMLR %P 191--200 %U https://proceedings.mlr.press/v307/kampen26a.html %V 307 %X Accurate segmentation is essential in error-critical domains such as medical imaging, where outputs support clinical decisions. Probabilistic models like the Stochastic Segmentation Network (SSN) enable uncertainty quantification, but existing methods typically use low-rank plus diagonal covariance structures that struggle to capture both global and local spatial correlations, limiting performance gains over deterministic models. We revisit low-rank formulations and introduce two approaches - Single-Basis and Mixture-of-Bases decompositions - that project predicted noise onto learned covariance bases, either globally or within partitioned volume blocks. This yields richer, more flexible uncertainty modeling with minimal parameter overhead. On the most challenging organs in the 3D TotalSegmentator CT dataset, our methods significantly improve Dice scores over deterministic and baseline stochastic models while preserving strong calibration, with the Mixture-of-Bases performing best. These findings show that basis-driven covariance modeling can enhance segmentation accuracy and uncertainty estimation in 3D medical imaging.
APA
Kampen, P.J., Aspe, A.W., Juhl, K.A., Christensen, A.N., Hannemose, M.R., Dahl, A., Paulsen, R.R. & Sundgaard, J.V.. (2026). Structured Covariance Modeling Using Learned Mixture-of-Bases for Uncertainty in 3D Segmentation. Proceedings of the 7th Northern Lights Deep Learning Conference (NLDL), in Proceedings of Machine Learning Research 307:191-200 Available from https://proceedings.mlr.press/v307/kampen26a.html.

Related Material