Zero-Shot Rankability: Revealing Latent Ordinal Structure in Multimodal Large Language Models via Language

Nam Hyeon-Woo, Moon Ye-Bin, Sohwi Lim, Kwon Byung-Ki, Tae-Hyun Oh
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:49263-49280, 2026.

Abstract

Recent work shows that vision encoders capture ordinal attributes along linear axes, which can be recovered from as few as two labeled images. However, in the zero-shot setting, the text-driven rank axis for Vision-Language Models (VLMs) like CLIP remains suboptimal. In this work, we study the embeddings of Multimodal LLMs (MLLMs). We hypothesize that MLLMs can overcome this limitation due to three potential advantages: their inherent ordinal understanding, capacity for conditional embeddings, and a small cross-modal gap. We show that MLLMs are rankable using only text prompts. Experiments demonstrate that a text-driven rank axis for MLLM embeddings achieves 90% of the performance of the supervised linear rank axis, significantly outperforming the 61% observed in VLM embeddings. We validate that this capability stems from MLLMs’ conditional embeddings and a smaller modality gap than VLMs. Furthermore, we demonstrate that this property generalizes to the audio domain. Our findings suggest that language provides a direct interface for probing latent ordinal structures in MLLMs. Code and prompts are available at https://github.com/kaist-ami/prompt-probing

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-hyeon-woo26a, title = {Zero-Shot Rankability: Revealing Latent Ordinal Structure in Multimodal Large Language Models via Language}, author = {Hyeon-Woo, Nam and Ye-Bin, Moon and Lim, Sohwi and Byung-Ki, Kwon and Oh, Tae-Hyun}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {49263--49280}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/hyeon-woo26a/hyeon-woo26a.pdf}, url = {https://proceedings.mlr.press/v306/hyeon-woo26a.html}, abstract = {Recent work shows that vision encoders capture ordinal attributes along linear axes, which can be recovered from as few as two labeled images. However, in the zero-shot setting, the text-driven rank axis for Vision-Language Models (VLMs) like CLIP remains suboptimal. In this work, we study the embeddings of Multimodal LLMs (MLLMs). We hypothesize that MLLMs can overcome this limitation due to three potential advantages: their inherent ordinal understanding, capacity for conditional embeddings, and a small cross-modal gap. We show that MLLMs are rankable using only text prompts. Experiments demonstrate that a text-driven rank axis for MLLM embeddings achieves 90% of the performance of the supervised linear rank axis, significantly outperforming the 61% observed in VLM embeddings. We validate that this capability stems from MLLMs’ conditional embeddings and a smaller modality gap than VLMs. Furthermore, we demonstrate that this property generalizes to the audio domain. Our findings suggest that language provides a direct interface for probing latent ordinal structures in MLLMs. Code and prompts are available at https://github.com/kaist-ami/prompt-probing} }
Endnote
%0 Conference Paper %T Zero-Shot Rankability: Revealing Latent Ordinal Structure in Multimodal Large Language Models via Language %A Nam Hyeon-Woo %A Moon Ye-Bin %A Sohwi Lim %A Kwon Byung-Ki %A Tae-Hyun Oh %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-hyeon-woo26a %I PMLR %P 49263--49280 %U https://proceedings.mlr.press/v306/hyeon-woo26a.html %V 306 %X Recent work shows that vision encoders capture ordinal attributes along linear axes, which can be recovered from as few as two labeled images. However, in the zero-shot setting, the text-driven rank axis for Vision-Language Models (VLMs) like CLIP remains suboptimal. In this work, we study the embeddings of Multimodal LLMs (MLLMs). We hypothesize that MLLMs can overcome this limitation due to three potential advantages: their inherent ordinal understanding, capacity for conditional embeddings, and a small cross-modal gap. We show that MLLMs are rankable using only text prompts. Experiments demonstrate that a text-driven rank axis for MLLM embeddings achieves 90% of the performance of the supervised linear rank axis, significantly outperforming the 61% observed in VLM embeddings. We validate that this capability stems from MLLMs’ conditional embeddings and a smaller modality gap than VLMs. Furthermore, we demonstrate that this property generalizes to the audio domain. Our findings suggest that language provides a direct interface for probing latent ordinal structures in MLLMs. Code and prompts are available at https://github.com/kaist-ami/prompt-probing
APA
Hyeon-Woo, N., Ye-Bin, M., Lim, S., Byung-Ki, K. & Oh, T.. (2026). Zero-Shot Rankability: Revealing Latent Ordinal Structure in Multimodal Large Language Models via Language. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:49263-49280 Available from https://proceedings.mlr.press/v306/hyeon-woo26a.html.

Related Material