MedMosaic: A Challenging Large Scale Benchmark of Diverse Medical Audio

Harshit Rajgarhia, Shuubham Ojha, Asif Shaik, Akhil Pothanapalli, Rachuri Lokesh, Abhishek Mukherji, Prasanna Desikan
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:103261-103320, 2026.

Abstract

Medical audio data is difficult to collect due to privacy regulations and high annotation costs arising from domain expertise. Thus, existing benchmarks tend to underrepresent complex medical audio scenarios. To address this challenge, we present MedMosaic, a medical audio question–answering dataset designed to benchmark language and audio reasoning models under realistic clinical constraints. MedMosaic features a diverse range of medical audio types, including condition-related physiological sounds, carefully constructed synthetic voices to mimic speech with artifacts as well as real short and long length clinical conversations to model varying context lengths. The dataset also features a total of 46,701 question-answer pairs, spanning categories such as multiple-choice, sequential multi-turn, and open-ended question–answers, enabling systematic evaluation of multi-hop reasoning and answer generation capabilities. Benchmarking 13 audio and multimodal reasoning models reveals that reasoning remains challenging for all evaluated systems, with substantial performance variation across question types. In particular, even state-of-the-art model like Gemini-2.5-pro can only achieve 68.1% accuracy approximately. These findings underscore persistent limitations in medical reasoning and highlight the need for more robust, domain-specific multimodal reasoning models. A sample of benchmark data is available here:https://shorturl.at/Lyp33

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-rajgarhia26a, title = {{M}ed{M}osaic: A Challenging Large Scale Benchmark of Diverse Medical Audio}, author = {Rajgarhia, Harshit and Ojha, Shuubham and Shaik, Asif and Pothanapalli, Akhil and Lokesh, Rachuri and Mukherji, Abhishek and Desikan, Prasanna}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {103261--103320}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/rajgarhia26a/rajgarhia26a.pdf}, url = {https://proceedings.mlr.press/v306/rajgarhia26a.html}, abstract = {Medical audio data is difficult to collect due to privacy regulations and high annotation costs arising from domain expertise. Thus, existing benchmarks tend to underrepresent complex medical audio scenarios. To address this challenge, we present MedMosaic, a medical audio question–answering dataset designed to benchmark language and audio reasoning models under realistic clinical constraints. MedMosaic features a diverse range of medical audio types, including condition-related physiological sounds, carefully constructed synthetic voices to mimic speech with artifacts as well as real short and long length clinical conversations to model varying context lengths. The dataset also features a total of 46,701 question-answer pairs, spanning categories such as multiple-choice, sequential multi-turn, and open-ended question–answers, enabling systematic evaluation of multi-hop reasoning and answer generation capabilities. Benchmarking 13 audio and multimodal reasoning models reveals that reasoning remains challenging for all evaluated systems, with substantial performance variation across question types. In particular, even state-of-the-art model like Gemini-2.5-pro can only achieve 68.1% accuracy approximately. These findings underscore persistent limitations in medical reasoning and highlight the need for more robust, domain-specific multimodal reasoning models. A sample of benchmark data is available here:https://shorturl.at/Lyp33} }
Endnote
%0 Conference Paper %T MedMosaic: A Challenging Large Scale Benchmark of Diverse Medical Audio %A Harshit Rajgarhia %A Shuubham Ojha %A Asif Shaik %A Akhil Pothanapalli %A Rachuri Lokesh %A Abhishek Mukherji %A Prasanna Desikan %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-rajgarhia26a %I PMLR %P 103261--103320 %U https://proceedings.mlr.press/v306/rajgarhia26a.html %V 306 %X Medical audio data is difficult to collect due to privacy regulations and high annotation costs arising from domain expertise. Thus, existing benchmarks tend to underrepresent complex medical audio scenarios. To address this challenge, we present MedMosaic, a medical audio question–answering dataset designed to benchmark language and audio reasoning models under realistic clinical constraints. MedMosaic features a diverse range of medical audio types, including condition-related physiological sounds, carefully constructed synthetic voices to mimic speech with artifacts as well as real short and long length clinical conversations to model varying context lengths. The dataset also features a total of 46,701 question-answer pairs, spanning categories such as multiple-choice, sequential multi-turn, and open-ended question–answers, enabling systematic evaluation of multi-hop reasoning and answer generation capabilities. Benchmarking 13 audio and multimodal reasoning models reveals that reasoning remains challenging for all evaluated systems, with substantial performance variation across question types. In particular, even state-of-the-art model like Gemini-2.5-pro can only achieve 68.1% accuracy approximately. These findings underscore persistent limitations in medical reasoning and highlight the need for more robust, domain-specific multimodal reasoning models. A sample of benchmark data is available here:https://shorturl.at/Lyp33
APA
Rajgarhia, H., Ojha, S., Shaik, A., Pothanapalli, A., Lokesh, R., Mukherji, A. & Desikan, P.. (2026). MedMosaic: A Challenging Large Scale Benchmark of Diverse Medical Audio. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:103261-103320 Available from https://proceedings.mlr.press/v306/rajgarhia26a.html.

Related Material