Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs

Fei Wei, Daoyuan Chen, Ce Wang, Yilun Huang, Yushuo Chen, Xuchen Pan, Yaliang Li, Bolin Ding
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:133500-133523, 2026.

Abstract

Large language models (LLMs) are strong passive responders, but learning to proactively elicit information—asking the right questions and stopping at the right time—remains difficult. Existing approaches, such as optimizing turn-level attributes or relying on user simulators to generate training trajectories, often struggle with a persistent reality gap. We propose Learn-to-Ask, a simulator-free framework that learns proactive questioning policies directly from offline expert conversations. Our key insight is to leverage the observed future of each expert trajectory to derive dense, turn-level rewards that reflect expert long-horizon strategy, reducing policy learning to a sequence of supervised learning tasks that jointly enable LLMs to know what to ask and when to stop. To ensure the LLM-generated contents, such as reward fidelity and sampling quality, align with expectations, we further introduce an automated pipeline that calibrates the prompts with minimal human supervision. Across multiple datasets and model scales, Learn-to-Ask consistently improves proactive information-seeking behavior. We also report a large-scale real-world deployment where the trained agent surpasses an internal expert baseline under professional audit, which demonstrates the effectiveness of our framework and our rewards as a reality-validated proxy metric for LLM proactivity.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-wei26b, title = {Grounded in Reality: Learning and Deploying Proactive {LLM} from Offline Logs}, author = {Wei, Fei and Chen, Daoyuan and Wang, Ce and Huang, Yilun and Chen, Yushuo and Pan, Xuchen and Li, Yaliang and Ding, Bolin}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {133500--133523}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/wei26b/wei26b.pdf}, url = {https://proceedings.mlr.press/v306/wei26b.html}, abstract = {Large language models (LLMs) are strong passive responders, but learning to proactively elicit information—asking the right questions and stopping at the right time—remains difficult. Existing approaches, such as optimizing turn-level attributes or relying on user simulators to generate training trajectories, often struggle with a persistent reality gap. We propose Learn-to-Ask, a simulator-free framework that learns proactive questioning policies directly from offline expert conversations. Our key insight is to leverage the observed future of each expert trajectory to derive dense, turn-level rewards that reflect expert long-horizon strategy, reducing policy learning to a sequence of supervised learning tasks that jointly enable LLMs to know what to ask and when to stop. To ensure the LLM-generated contents, such as reward fidelity and sampling quality, align with expectations, we further introduce an automated pipeline that calibrates the prompts with minimal human supervision. Across multiple datasets and model scales, Learn-to-Ask consistently improves proactive information-seeking behavior. We also report a large-scale real-world deployment where the trained agent surpasses an internal expert baseline under professional audit, which demonstrates the effectiveness of our framework and our rewards as a reality-validated proxy metric for LLM proactivity.} }
Endnote
%0 Conference Paper %T Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs %A Fei Wei %A Daoyuan Chen %A Ce Wang %A Yilun Huang %A Yushuo Chen %A Xuchen Pan %A Yaliang Li %A Bolin Ding %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-wei26b %I PMLR %P 133500--133523 %U https://proceedings.mlr.press/v306/wei26b.html %V 306 %X Large language models (LLMs) are strong passive responders, but learning to proactively elicit information—asking the right questions and stopping at the right time—remains difficult. Existing approaches, such as optimizing turn-level attributes or relying on user simulators to generate training trajectories, often struggle with a persistent reality gap. We propose Learn-to-Ask, a simulator-free framework that learns proactive questioning policies directly from offline expert conversations. Our key insight is to leverage the observed future of each expert trajectory to derive dense, turn-level rewards that reflect expert long-horizon strategy, reducing policy learning to a sequence of supervised learning tasks that jointly enable LLMs to know what to ask and when to stop. To ensure the LLM-generated contents, such as reward fidelity and sampling quality, align with expectations, we further introduce an automated pipeline that calibrates the prompts with minimal human supervision. Across multiple datasets and model scales, Learn-to-Ask consistently improves proactive information-seeking behavior. We also report a large-scale real-world deployment where the trained agent surpasses an internal expert baseline under professional audit, which demonstrates the effectiveness of our framework and our rewards as a reality-validated proxy metric for LLM proactivity.
APA
Wei, F., Chen, D., Wang, C., Huang, Y., Chen, Y., Pan, X., Li, Y. & Ding, B.. (2026). Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:133500-133523 Available from https://proceedings.mlr.press/v306/wei26b.html.

Related Material