Video Prediction via Example Guidance

Jingwei Xu, Huazhe Xu, Bingbing Ni, Xiaokang Yang, Trevor Darrell
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:10628-10637, 2020.

Abstract

In video prediction tasks, one major challenge is to capture the multi-modal nature of future contents and dynamics. In this work, we propose a simple yet effective framework that can efficiently predict plausible future states, where the key insight is that the potential distribution of a sequence could be approximated with analogous ones in a repertoire of training pool, namely, expert examples. By further incorporating a novel optimization scheme into the training procedure, plausible predictions can be sampled efficiently from distribution constructed from the retrieved examples. Meanwhile, our method could be seamlessly integrated with existing stochastic predictive models; significant enhancement is observed with comprehensive experiments in both quantitative and qualitative aspects. We also demonstrate the generalization ability to predict the motion of unseen class, i.e., without access to corresponding data during training phase. Project Page: \hyperlink{https://sites.google.com/view/vpeg-supp/home.}{https://sites.google.com/view/vpeg-supp/home.}

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-xu20j, title = {Video Prediction via Example Guidance}, author = {Xu, Jingwei and Xu, Huazhe and Ni, Bingbing and Yang, Xiaokang and Darrell, Trevor}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {10628--10637}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/xu20j/xu20j.pdf}, url = {https://proceedings.mlr.press/v119/xu20j.html}, abstract = {In video prediction tasks, one major challenge is to capture the multi-modal nature of future contents and dynamics. In this work, we propose a simple yet effective framework that can efficiently predict plausible future states, where the key insight is that the potential distribution of a sequence could be approximated with analogous ones in a repertoire of training pool, namely, expert examples. By further incorporating a novel optimization scheme into the training procedure, plausible predictions can be sampled efficiently from distribution constructed from the retrieved examples. Meanwhile, our method could be seamlessly integrated with existing stochastic predictive models; significant enhancement is observed with comprehensive experiments in both quantitative and qualitative aspects. We also demonstrate the generalization ability to predict the motion of unseen class, i.e., without access to corresponding data during training phase. Project Page: \hyperlink{https://sites.google.com/view/vpeg-supp/home.}{https://sites.google.com/view/vpeg-supp/home.}} }
Endnote
%0 Conference Paper %T Video Prediction via Example Guidance %A Jingwei Xu %A Huazhe Xu %A Bingbing Ni %A Xiaokang Yang %A Trevor Darrell %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-xu20j %I PMLR %P 10628--10637 %U https://proceedings.mlr.press/v119/xu20j.html %V 119 %X In video prediction tasks, one major challenge is to capture the multi-modal nature of future contents and dynamics. In this work, we propose a simple yet effective framework that can efficiently predict plausible future states, where the key insight is that the potential distribution of a sequence could be approximated with analogous ones in a repertoire of training pool, namely, expert examples. By further incorporating a novel optimization scheme into the training procedure, plausible predictions can be sampled efficiently from distribution constructed from the retrieved examples. Meanwhile, our method could be seamlessly integrated with existing stochastic predictive models; significant enhancement is observed with comprehensive experiments in both quantitative and qualitative aspects. We also demonstrate the generalization ability to predict the motion of unseen class, i.e., without access to corresponding data during training phase. Project Page: \hyperlink{https://sites.google.com/view/vpeg-supp/home.}{https://sites.google.com/view/vpeg-supp/home.}
APA
Xu, J., Xu, H., Ni, B., Yang, X. & Darrell, T.. (2020). Video Prediction via Example Guidance. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:10628-10637 Available from https://proceedings.mlr.press/v119/xu20j.html.

Related Material