Embodied-DETR: End-to-End Temporal 3D Object Detection in Egocentric Views

Ziheng Ding, Xiaze Zhang, Yuejie Zhang, Lifeng Chen, Rui Feng
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:25333-25350, 2026.

Abstract

Embodied 3D object detection is a fundamental perceptual capability for embodied agents, in which observations are partial, heavily occluded, and sequential, requiring modeling of temporal continuity. However, existing benchmarks and methods are primarily designed for fully reconstructed global scenes and fail to capture temporal observation context and instance evolution in first-person perception. We introduce Embodied-Det, a new benchmark for embodied 3D object detection that evaluates detection accuracy, temporal stability, and consistency under egocentric sequential views. Building on this benchmark, we propose Embodied-DETR, an end-to-end temporal detection framework that models scene-level context and instance-level consistency through two complementary temporal modules, Scene-aware Feature Aggregation and Instance-aware Query Embedding. Experiments on Embodied-Det show that existing methods suffer substantial performance degradation in egocentric temporal settings, while Embodied-DETR achieves superior accuracy and temporal consistency, demonstrating the effectiveness of temporal modeling for embodied 3D perception. Codes are available at https://github.com/UniPerceptor/UniPerceptor.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-ding26x, title = {Embodied-{DETR}: End-to-End Temporal 3{D} Object Detection in Egocentric Views}, author = {Ding, Ziheng and Zhang, Xiaze and Zhang, Yuejie and Chen, Lifeng and Feng, Rui}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {25333--25350}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/ding26x/ding26x.pdf}, url = {https://proceedings.mlr.press/v306/ding26x.html}, abstract = {Embodied 3D object detection is a fundamental perceptual capability for embodied agents, in which observations are partial, heavily occluded, and sequential, requiring modeling of temporal continuity. However, existing benchmarks and methods are primarily designed for fully reconstructed global scenes and fail to capture temporal observation context and instance evolution in first-person perception. We introduce Embodied-Det, a new benchmark for embodied 3D object detection that evaluates detection accuracy, temporal stability, and consistency under egocentric sequential views. Building on this benchmark, we propose Embodied-DETR, an end-to-end temporal detection framework that models scene-level context and instance-level consistency through two complementary temporal modules, Scene-aware Feature Aggregation and Instance-aware Query Embedding. Experiments on Embodied-Det show that existing methods suffer substantial performance degradation in egocentric temporal settings, while Embodied-DETR achieves superior accuracy and temporal consistency, demonstrating the effectiveness of temporal modeling for embodied 3D perception. Codes are available at https://github.com/UniPerceptor/UniPerceptor.} }
Endnote
%0 Conference Paper %T Embodied-DETR: End-to-End Temporal 3D Object Detection in Egocentric Views %A Ziheng Ding %A Xiaze Zhang %A Yuejie Zhang %A Lifeng Chen %A Rui Feng %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-ding26x %I PMLR %P 25333--25350 %U https://proceedings.mlr.press/v306/ding26x.html %V 306 %X Embodied 3D object detection is a fundamental perceptual capability for embodied agents, in which observations are partial, heavily occluded, and sequential, requiring modeling of temporal continuity. However, existing benchmarks and methods are primarily designed for fully reconstructed global scenes and fail to capture temporal observation context and instance evolution in first-person perception. We introduce Embodied-Det, a new benchmark for embodied 3D object detection that evaluates detection accuracy, temporal stability, and consistency under egocentric sequential views. Building on this benchmark, we propose Embodied-DETR, an end-to-end temporal detection framework that models scene-level context and instance-level consistency through two complementary temporal modules, Scene-aware Feature Aggregation and Instance-aware Query Embedding. Experiments on Embodied-Det show that existing methods suffer substantial performance degradation in egocentric temporal settings, while Embodied-DETR achieves superior accuracy and temporal consistency, demonstrating the effectiveness of temporal modeling for embodied 3D perception. Codes are available at https://github.com/UniPerceptor/UniPerceptor.
APA
Ding, Z., Zhang, X., Zhang, Y., Chen, L. & Feng, R.. (2026). Embodied-DETR: End-to-End Temporal 3D Object Detection in Egocentric Views. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:25333-25350 Available from https://proceedings.mlr.press/v306/ding26x.html.

Related Material