$τ$-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains

Soham Ray, Keshav Dhandhania, Victor Barres, Karthik R Narasimhan
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:104031-104067, 2026.

Abstract

Full-duplex voice agents—systems that listen and speak simultaneously—are rapidly moving from research to production. However, existing evaluations address conversational dynamics and task completion in isolation. We introduce $\tau$-Voice, a benchmark for evaluating voice agents on grounded tasks with real-world complexity: agents must navigate complex multi-turn conversations, adhere to domain policies, and interact with the environment. The framework extends $\tau$$^2$-bench into a novel voice agent benchmark combining verifiable completion of complex grounded tasks, full-duplex interaction, and realistic audio—enabling direct comparison between voice and text performance. A controllable and realistic voice user simulator provides diverse accents, realistic audio environments, and rich turn-taking dynamics; by decoupling simulation from wall-clock time, the user simulator can use the most capable LLM without real-time constraints. We evaluate task completion (pass@1) and voice interaction quality across 278 tasks: while GPT-5 (reasoning) achieves 85%, voice agents reach only 31–51% under clean conditions and 26–38% under realistic conditions with noise and diverse accents—retaining only 30–45% of text capability; qualitative analysis confirms 79–90% of failures stem from agent behavior, suggesting that observed failures primarily reflect agent behavior under our evaluation setup. $\tau$-Voice provides a reproducible testbed for measuring progress toward voice agents that are natural, conversational, and reliable.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-ray26a, title = {$τ$-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains}, author = {Ray, Soham and Dhandhania, Keshav and Barres, Victor and Narasimhan, Karthik R}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {104031--104067}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/ray26a/ray26a.pdf}, url = {https://proceedings.mlr.press/v306/ray26a.html}, abstract = {Full-duplex voice agents—systems that listen and speak simultaneously—are rapidly moving from research to production. However, existing evaluations address conversational dynamics and task completion in isolation. We introduce $\tau$-Voice, a benchmark for evaluating voice agents on grounded tasks with real-world complexity: agents must navigate complex multi-turn conversations, adhere to domain policies, and interact with the environment. The framework extends $\tau$$^2$-bench into a novel voice agent benchmark combining verifiable completion of complex grounded tasks, full-duplex interaction, and realistic audio—enabling direct comparison between voice and text performance. A controllable and realistic voice user simulator provides diverse accents, realistic audio environments, and rich turn-taking dynamics; by decoupling simulation from wall-clock time, the user simulator can use the most capable LLM without real-time constraints. We evaluate task completion (pass@1) and voice interaction quality across 278 tasks: while GPT-5 (reasoning) achieves 85%, voice agents reach only 31–51% under clean conditions and 26–38% under realistic conditions with noise and diverse accents—retaining only 30–45% of text capability; qualitative analysis confirms 79–90% of failures stem from agent behavior, suggesting that observed failures primarily reflect agent behavior under our evaluation setup. $\tau$-Voice provides a reproducible testbed for measuring progress toward voice agents that are natural, conversational, and reliable.} }
Endnote
%0 Conference Paper %T $τ$-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains %A Soham Ray %A Keshav Dhandhania %A Victor Barres %A Karthik R Narasimhan %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-ray26a %I PMLR %P 104031--104067 %U https://proceedings.mlr.press/v306/ray26a.html %V 306 %X Full-duplex voice agents—systems that listen and speak simultaneously—are rapidly moving from research to production. However, existing evaluations address conversational dynamics and task completion in isolation. We introduce $\tau$-Voice, a benchmark for evaluating voice agents on grounded tasks with real-world complexity: agents must navigate complex multi-turn conversations, adhere to domain policies, and interact with the environment. The framework extends $\tau$$^2$-bench into a novel voice agent benchmark combining verifiable completion of complex grounded tasks, full-duplex interaction, and realistic audio—enabling direct comparison between voice and text performance. A controllable and realistic voice user simulator provides diverse accents, realistic audio environments, and rich turn-taking dynamics; by decoupling simulation from wall-clock time, the user simulator can use the most capable LLM without real-time constraints. We evaluate task completion (pass@1) and voice interaction quality across 278 tasks: while GPT-5 (reasoning) achieves 85%, voice agents reach only 31–51% under clean conditions and 26–38% under realistic conditions with noise and diverse accents—retaining only 30–45% of text capability; qualitative analysis confirms 79–90% of failures stem from agent behavior, suggesting that observed failures primarily reflect agent behavior under our evaluation setup. $\tau$-Voice provides a reproducible testbed for measuring progress toward voice agents that are natural, conversational, and reliable.
APA
Ray, S., Dhandhania, K., Barres, V. & Narasimhan, K.R.. (2026). $τ$-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:104031-104067 Available from https://proceedings.mlr.press/v306/ray26a.html.

Related Material