A Diagnostic Study of Multi-Agent LLMs for Real-World Debates

Priya Pitre, Gaurav Srivastava, Lu Zhang, Le Wang, Naren Ramakrishnan, Xuan Wang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:99003-99017, 2026.

Abstract

Multi-agent LLM debates are increasingly used in domains such as policy, politics, and city planning, where ground truth is often unavailable. Yet existing evaluations rely heavily on outcome-based proxies such as consensus, majority vote, or LLM-as-judge scores, which can miss failures like sycophancy, domination, and premature convergence. We introduce a diagnostic framework that evaluates both debate outcomes and the deliberative process using interpretable metrics for engagement, responsiveness, influence asymmetry, balance, stability, and agent utility. Across real-world debate settings and validation benchmarks, our process-level diagnostics align more closely with human judgments and reveal interaction failures that standard outcome-only measures overlook. These results show that reliable evaluation of multi-agent debates requires measuring not only what answer agents reach, but how they reach it.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-pitre26a, title = {A Diagnostic Study of Multi-Agent {LLM}s for Real-World Debates}, author = {Pitre, Priya and Srivastava, Gaurav and Zhang, Lu and Wang, Le and Ramakrishnan, Naren and Wang, Xuan}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {99003--99017}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/pitre26a/pitre26a.pdf}, url = {https://proceedings.mlr.press/v306/pitre26a.html}, abstract = {Multi-agent LLM debates are increasingly used in domains such as policy, politics, and city planning, where ground truth is often unavailable. Yet existing evaluations rely heavily on outcome-based proxies such as consensus, majority vote, or LLM-as-judge scores, which can miss failures like sycophancy, domination, and premature convergence. We introduce a diagnostic framework that evaluates both debate outcomes and the deliberative process using interpretable metrics for engagement, responsiveness, influence asymmetry, balance, stability, and agent utility. Across real-world debate settings and validation benchmarks, our process-level diagnostics align more closely with human judgments and reveal interaction failures that standard outcome-only measures overlook. These results show that reliable evaluation of multi-agent debates requires measuring not only what answer agents reach, but how they reach it.} }
Endnote
%0 Conference Paper %T A Diagnostic Study of Multi-Agent LLMs for Real-World Debates %A Priya Pitre %A Gaurav Srivastava %A Lu Zhang %A Le Wang %A Naren Ramakrishnan %A Xuan Wang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-pitre26a %I PMLR %P 99003--99017 %U https://proceedings.mlr.press/v306/pitre26a.html %V 306 %X Multi-agent LLM debates are increasingly used in domains such as policy, politics, and city planning, where ground truth is often unavailable. Yet existing evaluations rely heavily on outcome-based proxies such as consensus, majority vote, or LLM-as-judge scores, which can miss failures like sycophancy, domination, and premature convergence. We introduce a diagnostic framework that evaluates both debate outcomes and the deliberative process using interpretable metrics for engagement, responsiveness, influence asymmetry, balance, stability, and agent utility. Across real-world debate settings and validation benchmarks, our process-level diagnostics align more closely with human judgments and reveal interaction failures that standard outcome-only measures overlook. These results show that reliable evaluation of multi-agent debates requires measuring not only what answer agents reach, but how they reach it.
APA
Pitre, P., Srivastava, G., Zhang, L., Wang, L., Ramakrishnan, N. & Wang, X.. (2026). A Diagnostic Study of Multi-Agent LLMs for Real-World Debates. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:99003-99017 Available from https://proceedings.mlr.press/v306/pitre26a.html.

Related Material