SegMaST: Mamba-based Spatio-Temporal Modeling to Improve Longitudinal Disease Detection and Segmentation

Aswathi Varma, Jonas Weidner, Laurin Lux, Cosmin Bercea, Mark Mühlau, Jan Kirschke, Benedikt Wiestler, Daniel Rueckert
Proceedings of The 9th International Conference on Medical Imaging with Deep Learning, PMLR 315:1492-1508, 2026.

Abstract

Longitudinal medical image segmentation is fundamental for quantifying disease progression and evaluating treatment efficacy. However, two critical challenges persist: First, methods that jointly segment baseline and follow-up images remain underexplored, often missing the contextual benefits of simultaneous assessment and lacking longitudinal consistency. Second, real-world datasets typically exhibit severe class imbalance, as scans showing actual disease progression are far rarer than those showing stable anatomy, an issue frequently neglected by existing models. To address these limitations, we propose SegMaST, a novel Mamba-based spatio-temporal framework. Unlike conventional approaches that treat timepoints in isolation, SegMaST leverages cross-temporal information and spatial correspondences to jointly segment the initial baseline mask and explicitly localize new or progressive pathologies in follow-up scans. Additionally, we introduce an imbalance-aware loss accumulation strategy to enhance robustness in realistic clinical settings. On longitudinal cohorts of patients with Multiple Sclerosis (MS) and glioma, SegMaST outperforms established CNN- and attention-based baselines for follow-up segmentation (mean follow-up Dice MS in-house 0.536, MSSEG-2 0.620, and glioma 0.631) and lesion detection (F1 in-house 0.688, MSSEG-2 0.723), while maintaining state-of-the-art accuracy in baseline segmentation (Dice: 0.617 MS, 0.844 glioma).

Cite this Paper

BibTeX
@InProceedings{pmlr-v315-varma26a, title = {SegMaST: Mamba-based Spatio-Temporal Modeling to Improve Longitudinal Disease Detection and Segmentation}, author = {Varma, Aswathi and Weidner, Jonas and Lux, Laurin and Bercea, Cosmin and M{\"{u}}hlau, Mark and Kirschke, Jan and Wiestler, Benedikt and Rueckert, Daniel}, booktitle = {Proceedings of The 9th International Conference on Medical Imaging with Deep Learning}, pages = {1492--1508}, year = {2026}, editor = {Huo, Yuankai and Gao, Mingchen and Kuo, Chang-Fu and Jin, Yueming and Deng, Ruining}, volume = {315}, series = {Proceedings of Machine Learning Research}, month = {08--10 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v315/main/assets/varma26a/varma26a.pdf}, url = {https://proceedings.mlr.press/v315/varma26a.html}, abstract = {Longitudinal medical image segmentation is fundamental for quantifying disease progression and evaluating treatment efficacy. However, two critical challenges persist: First, methods that jointly segment baseline and follow-up images remain underexplored, often missing the contextual benefits of simultaneous assessment and lacking longitudinal consistency. Second, real-world datasets typically exhibit severe class imbalance, as scans showing actual disease progression are far rarer than those showing stable anatomy, an issue frequently neglected by existing models. To address these limitations, we propose SegMaST, a novel Mamba-based spatio-temporal framework. Unlike conventional approaches that treat timepoints in isolation, SegMaST leverages cross-temporal information and spatial correspondences to jointly segment the initial baseline mask and explicitly localize new or progressive pathologies in follow-up scans. Additionally, we introduce an imbalance-aware loss accumulation strategy to enhance robustness in realistic clinical settings. On longitudinal cohorts of patients with Multiple Sclerosis (MS) and glioma, SegMaST outperforms established CNN- and attention-based baselines for follow-up segmentation (mean follow-up Dice MS in-house 0.536, MSSEG-2 0.620, and glioma 0.631) and lesion detection (F1 in-house 0.688, MSSEG-2 0.723), while maintaining state-of-the-art accuracy in baseline segmentation (Dice: 0.617 MS, 0.844 glioma).} }
Endnote
%0 Conference Paper %T SegMaST: Mamba-based Spatio-Temporal Modeling to Improve Longitudinal Disease Detection and Segmentation %A Aswathi Varma %A Jonas Weidner %A Laurin Lux %A Cosmin Bercea %A Mark Mühlau %A Jan Kirschke %A Benedikt Wiestler %A Daniel Rueckert %B Proceedings of The 9th International Conference on Medical Imaging with Deep Learning %C Proceedings of Machine Learning Research %D 2026 %E Yuankai Huo %E Mingchen Gao %E Chang-Fu Kuo %E Yueming Jin %E Ruining Deng %F pmlr-v315-varma26a %I PMLR %P 1492--1508 %U https://proceedings.mlr.press/v315/varma26a.html %V 315 %X Longitudinal medical image segmentation is fundamental for quantifying disease progression and evaluating treatment efficacy. However, two critical challenges persist: First, methods that jointly segment baseline and follow-up images remain underexplored, often missing the contextual benefits of simultaneous assessment and lacking longitudinal consistency. Second, real-world datasets typically exhibit severe class imbalance, as scans showing actual disease progression are far rarer than those showing stable anatomy, an issue frequently neglected by existing models. To address these limitations, we propose SegMaST, a novel Mamba-based spatio-temporal framework. Unlike conventional approaches that treat timepoints in isolation, SegMaST leverages cross-temporal information and spatial correspondences to jointly segment the initial baseline mask and explicitly localize new or progressive pathologies in follow-up scans. Additionally, we introduce an imbalance-aware loss accumulation strategy to enhance robustness in realistic clinical settings. On longitudinal cohorts of patients with Multiple Sclerosis (MS) and glioma, SegMaST outperforms established CNN- and attention-based baselines for follow-up segmentation (mean follow-up Dice MS in-house 0.536, MSSEG-2 0.620, and glioma 0.631) and lesion detection (F1 in-house 0.688, MSSEG-2 0.723), while maintaining state-of-the-art accuracy in baseline segmentation (Dice: 0.617 MS, 0.844 glioma).
APA
Varma, A., Weidner, J., Lux, L., Bercea, C., Mühlau, M., Kirschke, J., Wiestler, B. & Rueckert, D.. (2026). SegMaST: Mamba-based Spatio-Temporal Modeling to Improve Longitudinal Disease Detection and Segmentation. Proceedings of The 9th International Conference on Medical Imaging with Deep Learning, in Proceedings of Machine Learning Research 315:1492-1508 Available from https://proceedings.mlr.press/v315/varma26a.html.

Related Material