Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View

Jin Wang, Shichao Dong, Yapeng Zhu, Kelu Yao, Weidong Zhao, Chao Li, Ping Luo
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:50332-50352, 2024.

Abstract

Compositional reasoning capabilities are usually considered as fundamental skills to characterize human perception. Recent studies show that current Vision Language Models (VLMs) surprisingly lack sufficient knowledge with respect to such capabilities. To this end, we propose to thoroughly diagnose the composition representations encoded by VLMs, systematically revealing the potential cause for this weakness. Specifically, we propose evaluation methods from a novel game-theoretic view to assess the vulnerability of VLMs on different aspects of compositional understanding, e.g., relations and attributes. Extensive experimental results demonstrate and validate several insights to understand the incapabilities of VLMs on compositional reasoning, which provide useful and reliable guidance for future studies. The deliverables will be updated here.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-wang24n, title = {Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View}, author = {Wang, Jin and Dong, Shichao and Zhu, Yapeng and Yao, Kelu and Zhao, Weidong and Li, Chao and Luo, Ping}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {50332--50352}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/wang24n/wang24n.pdf}, url = {https://proceedings.mlr.press/v235/wang24n.html}, abstract = {Compositional reasoning capabilities are usually considered as fundamental skills to characterize human perception. Recent studies show that current Vision Language Models (VLMs) surprisingly lack sufficient knowledge with respect to such capabilities. To this end, we propose to thoroughly diagnose the composition representations encoded by VLMs, systematically revealing the potential cause for this weakness. Specifically, we propose evaluation methods from a novel game-theoretic view to assess the vulnerability of VLMs on different aspects of compositional understanding, e.g., relations and attributes. Extensive experimental results demonstrate and validate several insights to understand the incapabilities of VLMs on compositional reasoning, which provide useful and reliable guidance for future studies. The deliverables will be updated here.} }
Endnote
%0 Conference Paper %T Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View %A Jin Wang %A Shichao Dong %A Yapeng Zhu %A Kelu Yao %A Weidong Zhao %A Chao Li %A Ping Luo %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-wang24n %I PMLR %P 50332--50352 %U https://proceedings.mlr.press/v235/wang24n.html %V 235 %X Compositional reasoning capabilities are usually considered as fundamental skills to characterize human perception. Recent studies show that current Vision Language Models (VLMs) surprisingly lack sufficient knowledge with respect to such capabilities. To this end, we propose to thoroughly diagnose the composition representations encoded by VLMs, systematically revealing the potential cause for this weakness. Specifically, we propose evaluation methods from a novel game-theoretic view to assess the vulnerability of VLMs on different aspects of compositional understanding, e.g., relations and attributes. Extensive experimental results demonstrate and validate several insights to understand the incapabilities of VLMs on compositional reasoning, which provide useful and reliable guidance for future studies. The deliverables will be updated here.
APA
Wang, J., Dong, S., Zhu, Y., Yao, K., Zhao, W., Li, C. & Luo, P.. (2024). Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:50332-50352 Available from https://proceedings.mlr.press/v235/wang24n.html.

Related Material