PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs

Artem Dementyev, Wazeer Zulfikar, Sinan Hersek, Pascal Getreuer, Anurag Kumar, Vivek Kumar
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:23672-23696, 2026.

Abstract

Current multimodal large language models (LLMs) process audio as a mono stream, ignoring the rich spatial information essential for embodied AI. Conversely, existing spatial audio models are constrained to fixed microphone geometries, preventing their deployment across diverse devices. We present PhaseCoder, a transformer-only spatial audio encoder that is inherently agnostic to microphone geometry. By taking raw multichannel audio and microphone coordinates as inputs, PhaseCoder performs accurate localization and produces robust spatial embeddings. We demonstrate that the Gemma 3n LLM can be fine-tuned to process and reason over the "Spatial Audio Tokens" produced by our encoder. PhaseCoder achieves state-of-the-art results on microphone-invariant localization benchmarks and, for the first time, enables an LLM to perform complex spatial reasoning and targeted transcription tasks from an arbitrary microphone array.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-dementyev26a, title = {{P}hase{C}oder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal {LLM}s}, author = {Dementyev, Artem and Zulfikar, Wazeer and Hersek, Sinan and Getreuer, Pascal and Kumar, Anurag and Kumar, Vivek}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {23672--23696}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/dementyev26a/dementyev26a.pdf}, url = {https://proceedings.mlr.press/v306/dementyev26a.html}, abstract = {Current multimodal large language models (LLMs) process audio as a mono stream, ignoring the rich spatial information essential for embodied AI. Conversely, existing spatial audio models are constrained to fixed microphone geometries, preventing their deployment across diverse devices. We present PhaseCoder, a transformer-only spatial audio encoder that is inherently agnostic to microphone geometry. By taking raw multichannel audio and microphone coordinates as inputs, PhaseCoder performs accurate localization and produces robust spatial embeddings. We demonstrate that the Gemma 3n LLM can be fine-tuned to process and reason over the "Spatial Audio Tokens" produced by our encoder. PhaseCoder achieves state-of-the-art results on microphone-invariant localization benchmarks and, for the first time, enables an LLM to perform complex spatial reasoning and targeted transcription tasks from an arbitrary microphone array.} }
Endnote
%0 Conference Paper %T PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs %A Artem Dementyev %A Wazeer Zulfikar %A Sinan Hersek %A Pascal Getreuer %A Anurag Kumar %A Vivek Kumar %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-dementyev26a %I PMLR %P 23672--23696 %U https://proceedings.mlr.press/v306/dementyev26a.html %V 306 %X Current multimodal large language models (LLMs) process audio as a mono stream, ignoring the rich spatial information essential for embodied AI. Conversely, existing spatial audio models are constrained to fixed microphone geometries, preventing their deployment across diverse devices. We present PhaseCoder, a transformer-only spatial audio encoder that is inherently agnostic to microphone geometry. By taking raw multichannel audio and microphone coordinates as inputs, PhaseCoder performs accurate localization and produces robust spatial embeddings. We demonstrate that the Gemma 3n LLM can be fine-tuned to process and reason over the "Spatial Audio Tokens" produced by our encoder. PhaseCoder achieves state-of-the-art results on microphone-invariant localization benchmarks and, for the first time, enables an LLM to perform complex spatial reasoning and targeted transcription tasks from an arbitrary microphone array.
APA
Dementyev, A., Zulfikar, W., Hersek, S., Getreuer, P., Kumar, A. & Kumar, V.. (2026). PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:23672-23696 Available from https://proceedings.mlr.press/v306/dementyev26a.html.

Related Material