ProAct: A Benchmark and Multimodal Framework for Structure-Aware Proactive Response

Xiaomeng Zhu, Fengming Zhu, Weijie Zhou, Ye Tian, Zhenlin Hu, Yufei Huang, Yuchun Guo, Xinyu Wu, Zhengyou Zhang, Fangzhen Lin, Xuantang Xiong
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:168110-168137, 2026.

Abstract

While passive agents merely follow instructions, proactive agents align with higher-level objectives, such as assistance and safety by continuously monitoring the environment to determine when and how to act. However, developing proactive agents is hindered by the lack of specialized resources. To address this, we introduce ProAct-75, a benchmark designed to train and evaluate proactive agents across diverse domains, including assistance, maintenance, and safety monitoring. Spanning 75 tasks, our dataset features 91,581 step-level annotations enriched with explicit task graphs. These graphs encode step dependencies and parallel execution possibilities, providing the structural grounding necessary for complex decision-making. Building on this benchmark, we propose ProAct-Helper, a reference baseline powered by a Multimodal Large Language Model (MLLM) that grounds decision-making in state detection, and leveraging task graphs to enable entropy-driven heuristic search for action selection, allowing agents to execute parallel threads independently rather than mirroring the human’s next step. Extensive experiments demonstrate that ProAct-Helper outperforms strong closed-source models, improving trigger detection mF1 by 6.21%, saving 0.25 more steps in online one-step decision, and increasing the rate of parallel actions by 15.58%. Code is available at https://github.com/ZhuXMMM/ProAct.git

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-zhu26bj, title = {{P}ro{A}ct: A Benchmark and Multimodal Framework for Structure-Aware Proactive Response}, author = {Zhu, Xiaomeng and Zhu, Fengming and Zhou, Weijie and Tian, Ye and Hu, Zhenlin and Huang, Yufei and Guo, Yuchun and Wu, Xinyu and Zhang, Zhengyou and Lin, Fangzhen and Xiong, Xuantang}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {168110--168137}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/zhu26bj/zhu26bj.pdf}, url = {https://proceedings.mlr.press/v306/zhu26bj.html}, abstract = {While passive agents merely follow instructions, proactive agents align with higher-level objectives, such as assistance and safety by continuously monitoring the environment to determine when and how to act. However, developing proactive agents is hindered by the lack of specialized resources. To address this, we introduce ProAct-75, a benchmark designed to train and evaluate proactive agents across diverse domains, including assistance, maintenance, and safety monitoring. Spanning 75 tasks, our dataset features 91,581 step-level annotations enriched with explicit task graphs. These graphs encode step dependencies and parallel execution possibilities, providing the structural grounding necessary for complex decision-making. Building on this benchmark, we propose ProAct-Helper, a reference baseline powered by a Multimodal Large Language Model (MLLM) that grounds decision-making in state detection, and leveraging task graphs to enable entropy-driven heuristic search for action selection, allowing agents to execute parallel threads independently rather than mirroring the human’s next step. Extensive experiments demonstrate that ProAct-Helper outperforms strong closed-source models, improving trigger detection mF1 by 6.21%, saving 0.25 more steps in online one-step decision, and increasing the rate of parallel actions by 15.58%. Code is available at https://github.com/ZhuXMMM/ProAct.git} }
Endnote
%0 Conference Paper %T ProAct: A Benchmark and Multimodal Framework for Structure-Aware Proactive Response %A Xiaomeng Zhu %A Fengming Zhu %A Weijie Zhou %A Ye Tian %A Zhenlin Hu %A Yufei Huang %A Yuchun Guo %A Xinyu Wu %A Zhengyou Zhang %A Fangzhen Lin %A Xuantang Xiong %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-zhu26bj %I PMLR %P 168110--168137 %U https://proceedings.mlr.press/v306/zhu26bj.html %V 306 %X While passive agents merely follow instructions, proactive agents align with higher-level objectives, such as assistance and safety by continuously monitoring the environment to determine when and how to act. However, developing proactive agents is hindered by the lack of specialized resources. To address this, we introduce ProAct-75, a benchmark designed to train and evaluate proactive agents across diverse domains, including assistance, maintenance, and safety monitoring. Spanning 75 tasks, our dataset features 91,581 step-level annotations enriched with explicit task graphs. These graphs encode step dependencies and parallel execution possibilities, providing the structural grounding necessary for complex decision-making. Building on this benchmark, we propose ProAct-Helper, a reference baseline powered by a Multimodal Large Language Model (MLLM) that grounds decision-making in state detection, and leveraging task graphs to enable entropy-driven heuristic search for action selection, allowing agents to execute parallel threads independently rather than mirroring the human’s next step. Extensive experiments demonstrate that ProAct-Helper outperforms strong closed-source models, improving trigger detection mF1 by 6.21%, saving 0.25 more steps in online one-step decision, and increasing the rate of parallel actions by 15.58%. Code is available at https://github.com/ZhuXMMM/ProAct.git
APA
Zhu, X., Zhu, F., Zhou, W., Tian, Y., Hu, Z., Huang, Y., Guo, Y., Wu, X., Zhang, Z., Lin, F. & Xiong, X.. (2026). ProAct: A Benchmark and Multimodal Framework for Structure-Aware Proactive Response. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:168110-168137 Available from https://proceedings.mlr.press/v306/zhu26bj.html.

Related Material