FeatureEndo-4DGS: Real-Time Deformable Surgical Scene Reconstruction and Segmentation with 4D Gaussian Splatting

Kai Li, Junhao Wang, William Han, Ding Zhao
Proceedings of the Fifth Machine Learning for Health Symposium, PMLR 297:1218-1234, 2026.

Abstract

Minimally invasive surgery ({MIS}) requires high-fidelity, real-time visual feedback of dynamic and low-texture surgical scenes. To address these requirements, we introduce FeatureEndo-4DGS ({FE-4DGS}), the first real-time pipeline leveraging feature-distilled {4D} Gaussian Splatting for simultaneous reconstruction and semantic segmentation of deformable surgical environments. Unlike prior feature-distilled methods restricted to static scenes, and existing {4D} approaches that lack semantic integration, {FE-4DGS} seamlessly leverages pre-trained {2D} semantic embeddings to produce a unified {4D} representation—where semantics also deform with tissue motion. This unified approach enables the generation of real-time {RGB} and semantic outputs through a single, parallelized rasterization process. Despite the additional complexity from feature distillation, {FE-4DGS} sustains real-time rendering (287.95 {FPS}) with a compact footprint, achieves state-of-the-art rendering fidelity on EndoNeRF (39.1 {PSNR}) and SCARED (27.3 {PSNR}), and delivers competitive EndoVis18 segmentation, matching or exceeding strong {2D} baselines for binary segmentation tasks (0.93 {DSC}) and remaining competitive for multi-label segmentation (0.77 {DSC}).

Cite this Paper

BibTeX
@InProceedings{pmlr-v297-li26a, title = {{FeatureEndo-4DGS}: Real-Time Deformable Surgical Scene Reconstruction and Segmentation with {4D} Gaussian Splatting}, author = {Li, Kai and Wang, Junhao and Han, William and Zhao, Ding}, booktitle = {Proceedings of the Fifth Machine Learning for Health Symposium}, pages = {1218--1234}, year = {2026}, editor = {Argaw, Peniel and Zhang, Haoran and Jabbour, Sarah and Chandak, Payal and Ji, Jerry and Mukherjee, Sumit and Salaudeen, Olawale and Chang, Trenton and Healey, Elizabeth and Gröger, Fabian and Adibi, Amin and Hegselmann, Stefan and Wild, Benjamin and Noori, Ayush}, volume = {297}, series = {Proceedings of Machine Learning Research}, month = {13--14 Dec}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v297/main/assets/li26a/li26a.pdf}, url = {https://proceedings.mlr.press/v297/li26a.html}, abstract = {Minimally invasive surgery ({MIS}) requires high-fidelity, real-time visual feedback of dynamic and low-texture surgical scenes. To address these requirements, we introduce FeatureEndo-4DGS ({FE-4DGS}), the first real-time pipeline leveraging feature-distilled {4D} Gaussian Splatting for simultaneous reconstruction and semantic segmentation of deformable surgical environments. Unlike prior feature-distilled methods restricted to static scenes, and existing {4D} approaches that lack semantic integration, {FE-4DGS} seamlessly leverages pre-trained {2D} semantic embeddings to produce a unified {4D} representation—where semantics also deform with tissue motion. This unified approach enables the generation of real-time {RGB} and semantic outputs through a single, parallelized rasterization process. Despite the additional complexity from feature distillation, {FE-4DGS} sustains real-time rendering (287.95 {FPS}) with a compact footprint, achieves state-of-the-art rendering fidelity on EndoNeRF (39.1 {PSNR}) and SCARED (27.3 {PSNR}), and delivers competitive EndoVis18 segmentation, matching or exceeding strong {2D} baselines for binary segmentation tasks (0.93 {DSC}) and remaining competitive for multi-label segmentation (0.77 {DSC}).} }
Endnote
%0 Conference Paper %T FeatureEndo-4DGS: Real-Time Deformable Surgical Scene Reconstruction and Segmentation with 4D Gaussian Splatting %A Kai Li %A Junhao Wang %A William Han %A Ding Zhao %B Proceedings of the Fifth Machine Learning for Health Symposium %C Proceedings of Machine Learning Research %D 2026 %E Peniel Argaw %E Haoran Zhang %E Sarah Jabbour %E Payal Chandak %E Jerry Ji %E Sumit Mukherjee %E Olawale Salaudeen %E Trenton Chang %E Elizabeth Healey %E Fabian Gröger %E Amin Adibi %E Stefan Hegselmann %E Benjamin Wild %E Ayush Noori %F pmlr-v297-li26a %I PMLR %P 1218--1234 %U https://proceedings.mlr.press/v297/li26a.html %V 297 %X Minimally invasive surgery ({MIS}) requires high-fidelity, real-time visual feedback of dynamic and low-texture surgical scenes. To address these requirements, we introduce FeatureEndo-4DGS ({FE-4DGS}), the first real-time pipeline leveraging feature-distilled {4D} Gaussian Splatting for simultaneous reconstruction and semantic segmentation of deformable surgical environments. Unlike prior feature-distilled methods restricted to static scenes, and existing {4D} approaches that lack semantic integration, {FE-4DGS} seamlessly leverages pre-trained {2D} semantic embeddings to produce a unified {4D} representation—where semantics also deform with tissue motion. This unified approach enables the generation of real-time {RGB} and semantic outputs through a single, parallelized rasterization process. Despite the additional complexity from feature distillation, {FE-4DGS} sustains real-time rendering (287.95 {FPS}) with a compact footprint, achieves state-of-the-art rendering fidelity on EndoNeRF (39.1 {PSNR}) and SCARED (27.3 {PSNR}), and delivers competitive EndoVis18 segmentation, matching or exceeding strong {2D} baselines for binary segmentation tasks (0.93 {DSC}) and remaining competitive for multi-label segmentation (0.77 {DSC}).
APA
Li, K., Wang, J., Han, W. & Zhao, D.. (2026). FeatureEndo-4DGS: Real-Time Deformable Surgical Scene Reconstruction and Segmentation with 4D Gaussian Splatting. Proceedings of the Fifth Machine Learning for Health Symposium, in Proceedings of Machine Learning Research 297:1218-1234 Available from https://proceedings.mlr.press/v297/li26a.html.

Related Material