VideoPrism: A Foundational Visual Encoder for Video Understanding

Long Zhao, Nitesh Bharadwaj Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J. Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, Rachel Hornung, Florian Schroff, Ming-Hsuan Yang, David A Ross, Huisheng Wang, Hartwig Adam, Mikhail Sirotenko, Ting Liu, Boqing Gong
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:60785-60811, 2024.

Abstract

We introduce VideoPrism, a general-purpose video encoder that tackles diverse video understanding tasks with a single frozen model. We pretrain VideoPrism on a heterogeneous corpus containing 36M high-quality video-caption pairs and 582M video clips with noisy parallel text (e.g., ASR transcripts). The pretraining approach improves upon masked autoencoding by global-local distillation of semantic video embeddings and a token shuffling scheme, enabling VideoPrism to focus primarily on the video modality while leveraging the invaluable text associated with videos. We extensively test VideoPrism on four broad groups of video understanding tasks, from web video question answering to CV for science, achieving state-of-the-art performance on 31 out of 33 video understanding benchmarks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-zhao24f, title = {{V}ideo{P}rism: A Foundational Visual Encoder for Video Understanding}, author = {Zhao, Long and Gundavarapu, Nitesh Bharadwaj and Yuan, Liangzhe and Zhou, Hao and Yan, Shen and Sun, Jennifer J. and Friedman, Luke and Qian, Rui and Weyand, Tobias and Zhao, Yue and Hornung, Rachel and Schroff, Florian and Yang, Ming-Hsuan and Ross, David A and Wang, Huisheng and Adam, Hartwig and Sirotenko, Mikhail and Liu, Ting and Gong, Boqing}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {60785--60811}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/zhao24f/zhao24f.pdf}, url = {https://proceedings.mlr.press/v235/zhao24f.html}, abstract = {We introduce VideoPrism, a general-purpose video encoder that tackles diverse video understanding tasks with a single frozen model. We pretrain VideoPrism on a heterogeneous corpus containing 36M high-quality video-caption pairs and 582M video clips with noisy parallel text (e.g., ASR transcripts). The pretraining approach improves upon masked autoencoding by global-local distillation of semantic video embeddings and a token shuffling scheme, enabling VideoPrism to focus primarily on the video modality while leveraging the invaluable text associated with videos. We extensively test VideoPrism on four broad groups of video understanding tasks, from web video question answering to CV for science, achieving state-of-the-art performance on 31 out of 33 video understanding benchmarks.} }
Endnote
%0 Conference Paper %T VideoPrism: A Foundational Visual Encoder for Video Understanding %A Long Zhao %A Nitesh Bharadwaj Gundavarapu %A Liangzhe Yuan %A Hao Zhou %A Shen Yan %A Jennifer J. Sun %A Luke Friedman %A Rui Qian %A Tobias Weyand %A Yue Zhao %A Rachel Hornung %A Florian Schroff %A Ming-Hsuan Yang %A David A Ross %A Huisheng Wang %A Hartwig Adam %A Mikhail Sirotenko %A Ting Liu %A Boqing Gong %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-zhao24f %I PMLR %P 60785--60811 %U https://proceedings.mlr.press/v235/zhao24f.html %V 235 %X We introduce VideoPrism, a general-purpose video encoder that tackles diverse video understanding tasks with a single frozen model. We pretrain VideoPrism on a heterogeneous corpus containing 36M high-quality video-caption pairs and 582M video clips with noisy parallel text (e.g., ASR transcripts). The pretraining approach improves upon masked autoencoding by global-local distillation of semantic video embeddings and a token shuffling scheme, enabling VideoPrism to focus primarily on the video modality while leveraging the invaluable text associated with videos. We extensively test VideoPrism on four broad groups of video understanding tasks, from web video question answering to CV for science, achieving state-of-the-art performance on 31 out of 33 video understanding benchmarks.
APA
Zhao, L., Gundavarapu, N.B., Yuan, L., Zhou, H., Yan, S., Sun, J.J., Friedman, L., Qian, R., Weyand, T., Zhao, Y., Hornung, R., Schroff, F., Yang, M., Ross, D.A., Wang, H., Adam, H., Sirotenko, M., Liu, T. & Gong, B.. (2024). VideoPrism: A Foundational Visual Encoder for Video Understanding. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:60785-60811 Available from https://proceedings.mlr.press/v235/zhao24f.html.

Related Material