Multi-Round Human–AI Collaboration with User-Specified Requirements

Sima Noorani, Shayan Kiyani, Hamed Hassani, George J. Pappas
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:93747-93776, 2026.

Abstract

As humans increasingly rely on multi-round conversational AI for high-stakes decisions, principled frameworks are needed to ensure such interactions reliably improve decision quality. We adopt a human-centric view governed by two principles: counterfactual harm, ensuring the AI does not undermine human strengths, and complementarity, ensuring it adds value where the human is prone to err. We formalize these concepts via user-defined rules, allowing users to specify exactly what harm and complementarity mean for their specific task. We then introduce an online, distribution-free algorithm with finite-sample guarantees that enforces the user-specified constraints over the collaboration dynamics. We evaluate our framework across two interactive settings: LLM-simulated collaboration on a medical diagnostic task and a human crowdsourcing study on a pictorial reasoning task. We show that our online procedure maintains prescribed counterfactual-harm and complementarity violation rates even under non-stationary interaction dynamics. Moreover, tightening or loosening these constraints produces predictable shifts in downstream human accuracy, confirming that the two principles serve as practical levers for steering multi-round collaboration toward better decision quality without the need to model or constrain human behavior.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-noorani26a, title = {Multi-Round Human–{AI} Collaboration with User-Specified Requirements}, author = {Noorani, Sima and Kiyani, Shayan and Hassani, Hamed and Pappas, George J.}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {93747--93776}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/noorani26a/noorani26a.pdf}, url = {https://proceedings.mlr.press/v306/noorani26a.html}, abstract = {As humans increasingly rely on multi-round conversational AI for high-stakes decisions, principled frameworks are needed to ensure such interactions reliably improve decision quality. We adopt a human-centric view governed by two principles: counterfactual harm, ensuring the AI does not undermine human strengths, and complementarity, ensuring it adds value where the human is prone to err. We formalize these concepts via user-defined rules, allowing users to specify exactly what harm and complementarity mean for their specific task. We then introduce an online, distribution-free algorithm with finite-sample guarantees that enforces the user-specified constraints over the collaboration dynamics. We evaluate our framework across two interactive settings: LLM-simulated collaboration on a medical diagnostic task and a human crowdsourcing study on a pictorial reasoning task. We show that our online procedure maintains prescribed counterfactual-harm and complementarity violation rates even under non-stationary interaction dynamics. Moreover, tightening or loosening these constraints produces predictable shifts in downstream human accuracy, confirming that the two principles serve as practical levers for steering multi-round collaboration toward better decision quality without the need to model or constrain human behavior.} }
Endnote
%0 Conference Paper %T Multi-Round Human–AI Collaboration with User-Specified Requirements %A Sima Noorani %A Shayan Kiyani %A Hamed Hassani %A George J. Pappas %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-noorani26a %I PMLR %P 93747--93776 %U https://proceedings.mlr.press/v306/noorani26a.html %V 306 %X As humans increasingly rely on multi-round conversational AI for high-stakes decisions, principled frameworks are needed to ensure such interactions reliably improve decision quality. We adopt a human-centric view governed by two principles: counterfactual harm, ensuring the AI does not undermine human strengths, and complementarity, ensuring it adds value where the human is prone to err. We formalize these concepts via user-defined rules, allowing users to specify exactly what harm and complementarity mean for their specific task. We then introduce an online, distribution-free algorithm with finite-sample guarantees that enforces the user-specified constraints over the collaboration dynamics. We evaluate our framework across two interactive settings: LLM-simulated collaboration on a medical diagnostic task and a human crowdsourcing study on a pictorial reasoning task. We show that our online procedure maintains prescribed counterfactual-harm and complementarity violation rates even under non-stationary interaction dynamics. Moreover, tightening or loosening these constraints produces predictable shifts in downstream human accuracy, confirming that the two principles serve as practical levers for steering multi-round collaboration toward better decision quality without the need to model or constrain human behavior.
APA
Noorani, S., Kiyani, S., Hassani, H. & Pappas, G.J.. (2026). Multi-Round Human–AI Collaboration with User-Specified Requirements. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:93747-93776 Available from https://proceedings.mlr.press/v306/noorani26a.html.

Related Material