MODUS: Decoder-only Any-to-Any Modeling of Diverse Modalities

Mingqiao Ye, Zhaochong An, Zhitong Gao, Xian Liu, Oğuzhan Fatih Kar, Jesse Allardice, Roman Bachmann, David Mizrahi, François Fleuret, Chuan Li, Amir Zadeh, Serge Belongie, Afshin Dehghan, Amir Zamir
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:148139-148171, 2026.

Abstract

Any-to-any models predict any modality from any combination of others within a single network – a formulation used in multimodal vision and vision-language models, and increasingly in scientific domains such as ecology and astronomy. Existing any-to-any models are typically trained from scratch using encoder-decoder or diffusion architectures, impacting their performance and preventing them from using strong pre-trained decoder-only models as a prior. In this work, we investigate decoder-only any-to-any multimodal modeling, which treats all modalities symmetrically and supports arbitrary modalities as inputs and outputs without modality-specific heads, losses, or task pipelines. Because every modality is both an input and an output of the same model, the resulting model, named MODUS, can support a range of applications, such as chained generation through intermediate modalities or cross-modal self-verification by scoring the model’s own outputs with another generated modality. MODUS demonstrates strong out-of-the-box performance and is competitive with specialist and multitask baselines using a single model across various benchmarks. All materials are open-sourced.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-ye26a, title = {{MODUS}: Decoder-only Any-to-Any Modeling of Diverse Modalities}, author = {Ye, Mingqiao and An, Zhaochong and Gao, Zhitong and Liu, Xian and Kar, O\u{g}uzhan Fatih and Allardice, Jesse and Bachmann, Roman and Mizrahi, David and Fleuret, Fran\c{c}ois and Li, Chuan and Zadeh, Amir and Belongie, Serge and Dehghan, Afshin and Zamir, Amir}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {148139--148171}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/ye26a/ye26a.pdf}, url = {https://proceedings.mlr.press/v306/ye26a.html}, abstract = {Any-to-any models predict any modality from any combination of others within a single network – a formulation used in multimodal vision and vision-language models, and increasingly in scientific domains such as ecology and astronomy. Existing any-to-any models are typically trained from scratch using encoder-decoder or diffusion architectures, impacting their performance and preventing them from using strong pre-trained decoder-only models as a prior. In this work, we investigate decoder-only any-to-any multimodal modeling, which treats all modalities symmetrically and supports arbitrary modalities as inputs and outputs without modality-specific heads, losses, or task pipelines. Because every modality is both an input and an output of the same model, the resulting model, named MODUS, can support a range of applications, such as chained generation through intermediate modalities or cross-modal self-verification by scoring the model’s own outputs with another generated modality. MODUS demonstrates strong out-of-the-box performance and is competitive with specialist and multitask baselines using a single model across various benchmarks. All materials are open-sourced.} }
Endnote
%0 Conference Paper %T MODUS: Decoder-only Any-to-Any Modeling of Diverse Modalities %A Mingqiao Ye %A Zhaochong An %A Zhitong Gao %A Xian Liu %A Oğuzhan Fatih Kar %A Jesse Allardice %A Roman Bachmann %A David Mizrahi %A François Fleuret %A Chuan Li %A Amir Zadeh %A Serge Belongie %A Afshin Dehghan %A Amir Zamir %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-ye26a %I PMLR %P 148139--148171 %U https://proceedings.mlr.press/v306/ye26a.html %V 306 %X Any-to-any models predict any modality from any combination of others within a single network – a formulation used in multimodal vision and vision-language models, and increasingly in scientific domains such as ecology and astronomy. Existing any-to-any models are typically trained from scratch using encoder-decoder or diffusion architectures, impacting their performance and preventing them from using strong pre-trained decoder-only models as a prior. In this work, we investigate decoder-only any-to-any multimodal modeling, which treats all modalities symmetrically and supports arbitrary modalities as inputs and outputs without modality-specific heads, losses, or task pipelines. Because every modality is both an input and an output of the same model, the resulting model, named MODUS, can support a range of applications, such as chained generation through intermediate modalities or cross-modal self-verification by scoring the model’s own outputs with another generated modality. MODUS demonstrates strong out-of-the-box performance and is competitive with specialist and multitask baselines using a single model across various benchmarks. All materials are open-sourced.
APA
Ye, M., An, Z., Gao, Z., Liu, X., Kar, O.F., Allardice, J., Bachmann, R., Mizrahi, D., Fleuret, F., Li, C., Zadeh, A., Belongie, S., Dehghan, A. & Zamir, A.. (2026). MODUS: Decoder-only Any-to-Any Modeling of Diverse Modalities. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:148139-148171 Available from https://proceedings.mlr.press/v306/ye26a.html.

Related Material