Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts

Yan Zeng, Xinsong Zhang, Hang Li
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:25994-26009, 2022.

Abstract

Most existing methods in vision language pre-training rely on object-centric features extracted through object detection and make fine-grained alignments between the extracted features and texts. It is challenging for these methods to learn relations among multiple objects. To this end, we propose a new method called X-VLM to perform ‘multi-grained vision language pre-training.’ The key to learning multi-grained alignments is to locate visual concepts in the image given the associated texts, and in the meantime align the texts with the visual concepts, where the alignments are in multi-granularity. Experimental results show that X-VLM effectively leverages the learned multi-grained alignments to many downstream vision language tasks and consistently outperforms state-of-the-art methods.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-zeng22c, title = {Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts}, author = {Zeng, Yan and Zhang, Xinsong and Li, Hang}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {25994--26009}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/zeng22c/zeng22c.pdf}, url = {https://proceedings.mlr.press/v162/zeng22c.html}, abstract = {Most existing methods in vision language pre-training rely on object-centric features extracted through object detection and make fine-grained alignments between the extracted features and texts. It is challenging for these methods to learn relations among multiple objects. To this end, we propose a new method called X-VLM to perform ‘multi-grained vision language pre-training.’ The key to learning multi-grained alignments is to locate visual concepts in the image given the associated texts, and in the meantime align the texts with the visual concepts, where the alignments are in multi-granularity. Experimental results show that X-VLM effectively leverages the learned multi-grained alignments to many downstream vision language tasks and consistently outperforms state-of-the-art methods.} }
Endnote
%0 Conference Paper %T Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts %A Yan Zeng %A Xinsong Zhang %A Hang Li %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-zeng22c %I PMLR %P 25994--26009 %U https://proceedings.mlr.press/v162/zeng22c.html %V 162 %X Most existing methods in vision language pre-training rely on object-centric features extracted through object detection and make fine-grained alignments between the extracted features and texts. It is challenging for these methods to learn relations among multiple objects. To this end, we propose a new method called X-VLM to perform ‘multi-grained vision language pre-training.’ The key to learning multi-grained alignments is to locate visual concepts in the image given the associated texts, and in the meantime align the texts with the visual concepts, where the alignments are in multi-granularity. Experimental results show that X-VLM effectively leverages the learned multi-grained alignments to many downstream vision language tasks and consistently outperforms state-of-the-art methods.
APA
Zeng, Y., Zhang, X. & Li, H.. (2022). Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:25994-26009 Available from https://proceedings.mlr.press/v162/zeng22c.html.

Related Material