RVAS: Referring Video Active Exploration and Segmentation

Hengrui Hu, Weiwei Gao, Zipei Zhang, Henghui Ding
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:44993-45011, 2026.

Abstract

Existing referring video object segmentation (RVOS) is largely built on passive perception and assumes the target is already visible in the observed video, which limits real-world use when queries refer to objects beyond the current view. To address this gap, we introduce Referring Video Active Exploration and Segmentation (RVAS), a new task that focuses on reasoning about exploration policy and then locating and segmenting the object according to an input referring expression. To support RVAS, we build a large-scale dataset with manually annotated exploration actions and reference reasoning traces, enabling supervised training and evaluation. We benchmark representative RVOS and related video understanding baselines and find that they struggle to perform active target search and incur substantial overhead when coupled with online decision making. Motivated by these challenges, we propose LESA, a baseline framework that introduces a state controller and hierarchical memory for efficient streaming processing and sparse MLLM reasoning. LESA substantially reduces inference cost while maintaining competitive planning quality, and consistently improves segmentation accuracy on the RVAS dataset. Code is available at https://github.com/FudanCVL/RVAS.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-hu26q, title = {{RVAS}: Referring Video Active Exploration and Segmentation}, author = {Hu, Hengrui and Gao, Weiwei and Zhang, Zipei and Ding, Henghui}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {44993--45011}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/hu26q/hu26q.pdf}, url = {https://proceedings.mlr.press/v306/hu26q.html}, abstract = {Existing referring video object segmentation (RVOS) is largely built on passive perception and assumes the target is already visible in the observed video, which limits real-world use when queries refer to objects beyond the current view. To address this gap, we introduce Referring Video Active Exploration and Segmentation (RVAS), a new task that focuses on reasoning about exploration policy and then locating and segmenting the object according to an input referring expression. To support RVAS, we build a large-scale dataset with manually annotated exploration actions and reference reasoning traces, enabling supervised training and evaluation. We benchmark representative RVOS and related video understanding baselines and find that they struggle to perform active target search and incur substantial overhead when coupled with online decision making. Motivated by these challenges, we propose LESA, a baseline framework that introduces a state controller and hierarchical memory for efficient streaming processing and sparse MLLM reasoning. LESA substantially reduces inference cost while maintaining competitive planning quality, and consistently improves segmentation accuracy on the RVAS dataset. Code is available at https://github.com/FudanCVL/RVAS.} }
Endnote
%0 Conference Paper %T RVAS: Referring Video Active Exploration and Segmentation %A Hengrui Hu %A Weiwei Gao %A Zipei Zhang %A Henghui Ding %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-hu26q %I PMLR %P 44993--45011 %U https://proceedings.mlr.press/v306/hu26q.html %V 306 %X Existing referring video object segmentation (RVOS) is largely built on passive perception and assumes the target is already visible in the observed video, which limits real-world use when queries refer to objects beyond the current view. To address this gap, we introduce Referring Video Active Exploration and Segmentation (RVAS), a new task that focuses on reasoning about exploration policy and then locating and segmenting the object according to an input referring expression. To support RVAS, we build a large-scale dataset with manually annotated exploration actions and reference reasoning traces, enabling supervised training and evaluation. We benchmark representative RVOS and related video understanding baselines and find that they struggle to perform active target search and incur substantial overhead when coupled with online decision making. Motivated by these challenges, we propose LESA, a baseline framework that introduces a state controller and hierarchical memory for efficient streaming processing and sparse MLLM reasoning. LESA substantially reduces inference cost while maintaining competitive planning quality, and consistently improves segmentation accuracy on the RVAS dataset. Code is available at https://github.com/FudanCVL/RVAS.
APA
Hu, H., Gao, W., Zhang, Z. & Ding, H.. (2026). RVAS: Referring Video Active Exploration and Segmentation. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:44993-45011 Available from https://proceedings.mlr.press/v306/hu26q.html.

Related Material