VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation

Ming Dai, Sen Yang, Boqiang Duan, Boyuan Tong, Jiedong Zhuang, Wankou Yang, Jingdong Wang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:22590-22616, 2026.

Abstract

Reasoning Video Object Segmentation (RVOS) demands a sophisticated integration of temporal dynamics, spatial details, and linguistic reasoning to achieve precise pixel-level localization. Existing methods are limited to reasoning over fixed initial inputs and lack the capacity to actively acquire further visual evidence, which is often essential for resolving complex references in long or intricate videos. To address this, we propose $\textbf{VideoSEG-O3}$, the first multi-turn reinforcement learning framework for RVOS that emulates the human $\textit{“coarse-to-fine”}$ cognitive process. It employs a $\textit{multi-turn temporal-spatial chain-of-thought}$ to capture fine-grained details by iteratively pinpointing critical intervals and keyframes. Additionally, to enable the policy to perceive segmentation quality beyond mere text probability of $\texttt{[SEG]}$ during the RL stage, we introduce $\textit{SEG-aware logit calibration}$, which integrates pixel-wise segmentation feedback directly into the token-level logits. Furthermore, we design a $\textit{decoupled thinking trace}$ to hierarchically decompose the reasoning process into temporal, spatial, and linguistic dimensions, and construct $\textbf{VTS-CoT}$, a specialized cold-start dataset featuring comprehensive reasoning trajectories. Extensive experiments demonstrate that VideoSEG-O3 achieves advanced performance across 8 mainstream RVOS benchmarks, particularly excelling in long-horizon and complex reasoning tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-dai26j, title = {{V}ideo{SEG}-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation}, author = {Dai, Ming and Yang, Sen and Duan, Boqiang and Tong, Boyuan and Zhuang, Jiedong and Yang, Wankou and Wang, Jingdong}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {22590--22616}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/dai26j/dai26j.pdf}, url = {https://proceedings.mlr.press/v306/dai26j.html}, abstract = {Reasoning Video Object Segmentation (RVOS) demands a sophisticated integration of temporal dynamics, spatial details, and linguistic reasoning to achieve precise pixel-level localization. Existing methods are limited to reasoning over fixed initial inputs and lack the capacity to actively acquire further visual evidence, which is often essential for resolving complex references in long or intricate videos. To address this, we propose $\textbf{VideoSEG-O3}$, the first multi-turn reinforcement learning framework for RVOS that emulates the human $\textit{“coarse-to-fine”}$ cognitive process. It employs a $\textit{multi-turn temporal-spatial chain-of-thought}$ to capture fine-grained details by iteratively pinpointing critical intervals and keyframes. Additionally, to enable the policy to perceive segmentation quality beyond mere text probability of $\texttt{[SEG]}$ during the RL stage, we introduce $\textit{SEG-aware logit calibration}$, which integrates pixel-wise segmentation feedback directly into the token-level logits. Furthermore, we design a $\textit{decoupled thinking trace}$ to hierarchically decompose the reasoning process into temporal, spatial, and linguistic dimensions, and construct $\textbf{VTS-CoT}$, a specialized cold-start dataset featuring comprehensive reasoning trajectories. Extensive experiments demonstrate that VideoSEG-O3 achieves advanced performance across 8 mainstream RVOS benchmarks, particularly excelling in long-horizon and complex reasoning tasks.} }
Endnote
%0 Conference Paper %T VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation %A Ming Dai %A Sen Yang %A Boqiang Duan %A Boyuan Tong %A Jiedong Zhuang %A Wankou Yang %A Jingdong Wang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-dai26j %I PMLR %P 22590--22616 %U https://proceedings.mlr.press/v306/dai26j.html %V 306 %X Reasoning Video Object Segmentation (RVOS) demands a sophisticated integration of temporal dynamics, spatial details, and linguistic reasoning to achieve precise pixel-level localization. Existing methods are limited to reasoning over fixed initial inputs and lack the capacity to actively acquire further visual evidence, which is often essential for resolving complex references in long or intricate videos. To address this, we propose $\textbf{VideoSEG-O3}$, the first multi-turn reinforcement learning framework for RVOS that emulates the human $\textit{“coarse-to-fine”}$ cognitive process. It employs a $\textit{multi-turn temporal-spatial chain-of-thought}$ to capture fine-grained details by iteratively pinpointing critical intervals and keyframes. Additionally, to enable the policy to perceive segmentation quality beyond mere text probability of $\texttt{[SEG]}$ during the RL stage, we introduce $\textit{SEG-aware logit calibration}$, which integrates pixel-wise segmentation feedback directly into the token-level logits. Furthermore, we design a $\textit{decoupled thinking trace}$ to hierarchically decompose the reasoning process into temporal, spatial, and linguistic dimensions, and construct $\textbf{VTS-CoT}$, a specialized cold-start dataset featuring comprehensive reasoning trajectories. Extensive experiments demonstrate that VideoSEG-O3 achieves advanced performance across 8 mainstream RVOS benchmarks, particularly excelling in long-horizon and complex reasoning tasks.
APA
Dai, M., Yang, S., Duan, B., Tong, B., Zhuang, J., Yang, W. & Wang, J.. (2026). VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:22590-22616 Available from https://proceedings.mlr.press/v306/dai26j.html.

Related Material