Fast-SAM3D: 3Dfy Anything in Images but Faster

Weilun Feng, Mingqiang Wu, Zhiliang Chen, Chuanguang Yang, Haotong Qin, Yuqi Li, Xiaokun Liu, Guoxin Fan, Libo Huang, Yulun Zhang, Michele Magno, Yongjun Xu, Zhulin An
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:30470-30489, 2026.

Abstract

SAM3D enables scalable, open-world 3D reconstruction from complex scenes, yet its deployment is hindered by prohibitive inference latency. In this work, we conduct the first systematic investigation into its inference dynamics, revealing that generic acceleration strategies are brittle in this context. We demonstrate that these failures stem from neglecting the pipeline’s inherent multi-level heterogeneity: the kinematic distinctiveness between shape and layout, the intrinsic sparsity of texture refinement, and the spectral variance across geometries. To address this, we present Fast-SAM3D, a training-free framework that dynamically aligns computation with instantaneous generation complexity. Our approach integrates three heterogeneity-aware mechanisms: (1) Modality-Aware Step Caching to decouple structural evolution from sensitive layout updates; (2) Joint Spatiotemporal Token Carving to concentrate refinement on high-entropy regions; and (3) Spectral-Aware Token Aggregation to adapt decoding resolution. Extensive experiments demonstrate that Fast-SAM3D delivers up to 2.67$\times$ end-to-end speedup with negligible fidelity loss, establishing a new Pareto frontier for efficient single-view 3D generation.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-feng26t, title = {Fast-{SAM}3{D}: 3{D}fy Anything in Images but Faster}, author = {Feng, Weilun and Wu, Mingqiang and Chen, Zhiliang and Yang, Chuanguang and Qin, Haotong and Li, Yuqi and Liu, Xiaokun and Fan, Guoxin and Huang, Libo and Zhang, Yulun and Magno, Michele and Xu, Yongjun and An, Zhulin}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {30470--30489}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/feng26t/feng26t.pdf}, url = {https://proceedings.mlr.press/v306/feng26t.html}, abstract = {SAM3D enables scalable, open-world 3D reconstruction from complex scenes, yet its deployment is hindered by prohibitive inference latency. In this work, we conduct the first systematic investigation into its inference dynamics, revealing that generic acceleration strategies are brittle in this context. We demonstrate that these failures stem from neglecting the pipeline’s inherent multi-level heterogeneity: the kinematic distinctiveness between shape and layout, the intrinsic sparsity of texture refinement, and the spectral variance across geometries. To address this, we present Fast-SAM3D, a training-free framework that dynamically aligns computation with instantaneous generation complexity. Our approach integrates three heterogeneity-aware mechanisms: (1) Modality-Aware Step Caching to decouple structural evolution from sensitive layout updates; (2) Joint Spatiotemporal Token Carving to concentrate refinement on high-entropy regions; and (3) Spectral-Aware Token Aggregation to adapt decoding resolution. Extensive experiments demonstrate that Fast-SAM3D delivers up to 2.67$\times$ end-to-end speedup with negligible fidelity loss, establishing a new Pareto frontier for efficient single-view 3D generation.} }
Endnote
%0 Conference Paper %T Fast-SAM3D: 3Dfy Anything in Images but Faster %A Weilun Feng %A Mingqiang Wu %A Zhiliang Chen %A Chuanguang Yang %A Haotong Qin %A Yuqi Li %A Xiaokun Liu %A Guoxin Fan %A Libo Huang %A Yulun Zhang %A Michele Magno %A Yongjun Xu %A Zhulin An %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-feng26t %I PMLR %P 30470--30489 %U https://proceedings.mlr.press/v306/feng26t.html %V 306 %X SAM3D enables scalable, open-world 3D reconstruction from complex scenes, yet its deployment is hindered by prohibitive inference latency. In this work, we conduct the first systematic investigation into its inference dynamics, revealing that generic acceleration strategies are brittle in this context. We demonstrate that these failures stem from neglecting the pipeline’s inherent multi-level heterogeneity: the kinematic distinctiveness between shape and layout, the intrinsic sparsity of texture refinement, and the spectral variance across geometries. To address this, we present Fast-SAM3D, a training-free framework that dynamically aligns computation with instantaneous generation complexity. Our approach integrates three heterogeneity-aware mechanisms: (1) Modality-Aware Step Caching to decouple structural evolution from sensitive layout updates; (2) Joint Spatiotemporal Token Carving to concentrate refinement on high-entropy regions; and (3) Spectral-Aware Token Aggregation to adapt decoding resolution. Extensive experiments demonstrate that Fast-SAM3D delivers up to 2.67$\times$ end-to-end speedup with negligible fidelity loss, establishing a new Pareto frontier for efficient single-view 3D generation.
APA
Feng, W., Wu, M., Chen, Z., Yang, C., Qin, H., Li, Y., Liu, X., Fan, G., Huang, L., Zhang, Y., Magno, M., Xu, Y. & An, Z.. (2026). Fast-SAM3D: 3Dfy Anything in Images but Faster. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:30470-30489 Available from https://proceedings.mlr.press/v306/feng26t.html.

Related Material