Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streams

Yun Wang, Junbin Xiao, Han Lyu, Yifan Wang, Jing Zuo, Zhanjie Zhang, Hong Huang, Dapeng Wu, Angela Yao
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:131373-131417, 2026.

Abstract

We introduce UCS-Bench, a dataset spanning 170+ hours of egocentric visual observations with 8.1K+ timestamped questions for diagnosing User-Centric Continual Spatial intelligence in egocentric video streams. UCS-Bench targets a new problem that emphasizes dynamic spatial reasoning, long-term memory, and their alignment with users’ real-time locations. We propose DirectMe, a framework that incrementally constructs and maintains a structured spatial memory from streaming egocentric observations. DirectMe enables robust tracking and recall of object locations, all relative to the user’s movement over time. By tightly coupling visual perception with memory updates and spatial reasoning, our approach supports long-horizon queries that require recalling interactions, resolving viewpoint-induced ambiguities, and adapting to dynamic scenes. Our experiments show that DirectMe significantly improves the spatial reasoning of leading multimodal LLMs; it also surpasses many spatially aware and long-form streaming video models. We hope our benchmark and solution will advance spatial intelligence research for egocentric AI assistants. Data and code are available at https://github.com/cocowy1/UCS-Bench.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-wang26je, title = {Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streams}, author = {Wang, Yun and Xiao, Junbin and Lyu, Han and Wang, Yifan and Zuo, Jing and Zhang, Zhanjie and Huang, Hong and Wu, Dapeng and Yao, Angela}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {131373--131417}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/wang26je/wang26je.pdf}, url = {https://proceedings.mlr.press/v306/wang26je.html}, abstract = {We introduce UCS-Bench, a dataset spanning 170+ hours of egocentric visual observations with 8.1K+ timestamped questions for diagnosing User-Centric Continual Spatial intelligence in egocentric video streams. UCS-Bench targets a new problem that emphasizes dynamic spatial reasoning, long-term memory, and their alignment with users’ real-time locations. We propose DirectMe, a framework that incrementally constructs and maintains a structured spatial memory from streaming egocentric observations. DirectMe enables robust tracking and recall of object locations, all relative to the user’s movement over time. By tightly coupling visual perception with memory updates and spatial reasoning, our approach supports long-horizon queries that require recalling interactions, resolving viewpoint-induced ambiguities, and adapting to dynamic scenes. Our experiments show that DirectMe significantly improves the spatial reasoning of leading multimodal LLMs; it also surpasses many spatially aware and long-form streaming video models. We hope our benchmark and solution will advance spatial intelligence research for egocentric AI assistants. Data and code are available at https://github.com/cocowy1/UCS-Bench.} }
Endnote
%0 Conference Paper %T Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streams %A Yun Wang %A Junbin Xiao %A Han Lyu %A Yifan Wang %A Jing Zuo %A Zhanjie Zhang %A Hong Huang %A Dapeng Wu %A Angela Yao %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-wang26je %I PMLR %P 131373--131417 %U https://proceedings.mlr.press/v306/wang26je.html %V 306 %X We introduce UCS-Bench, a dataset spanning 170+ hours of egocentric visual observations with 8.1K+ timestamped questions for diagnosing User-Centric Continual Spatial intelligence in egocentric video streams. UCS-Bench targets a new problem that emphasizes dynamic spatial reasoning, long-term memory, and their alignment with users’ real-time locations. We propose DirectMe, a framework that incrementally constructs and maintains a structured spatial memory from streaming egocentric observations. DirectMe enables robust tracking and recall of object locations, all relative to the user’s movement over time. By tightly coupling visual perception with memory updates and spatial reasoning, our approach supports long-horizon queries that require recalling interactions, resolving viewpoint-induced ambiguities, and adapting to dynamic scenes. Our experiments show that DirectMe significantly improves the spatial reasoning of leading multimodal LLMs; it also surpasses many spatially aware and long-form streaming video models. We hope our benchmark and solution will advance spatial intelligence research for egocentric AI assistants. Data and code are available at https://github.com/cocowy1/UCS-Bench.
APA
Wang, Y., Xiao, J., Lyu, H., Wang, Y., Zuo, J., Zhang, Z., Huang, H., Wu, D. & Yao, A.. (2026). Keep It in Mind: User Centric Continual Spatial Intelligence Reasoning in Egocentric Video Streams. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:131373-131417 Available from https://proceedings.mlr.press/v306/wang26je.html.

Related Material