VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization

Andrei Atanov, Jesse Allardice, Roman Bachmann, Oğuzhan Fatih Kar, R Devon Hjelm, David Griffiths, Peter Fu, Afshin Dehghan, Amir Zamir
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:4259-4282, 2026.

Abstract

Visual tokenizers map high-dimensional raw pixels into a compressed representation for downstream modeling. Beyond compression, tokenizers dictate what information is preserved and how it is organized. A de facto standard approach to video tokenization is to represent a video as a spatiotemporal 3D grid of tokens, each capturing local information from the original signal. This requires the downstream model, e.g., a text-to-video model, to learn to predict all low-level details "pixel-by-pixel" irrespective of the video’s inherent complexity, leading to high learning complexity. We present VideoFlexTok, which represents videos with a variable-length sequence of tokens structured in a coarse-to-fine manner, where the first tokens (emergently) capture information such as semantics and motion, and later tokens add fine-grained details. The generative flow decoder enables realistic video reconstructions from any token count. This representation structure allows adapting the token count to downstream needs and encoding videos longer than the baselines within the same budget. We evaluate VideoFlexTok on class- and text-to-video generative tasks and show that it yields more efficient training than 3D grid tokens, achieving comparable generation quality (gFVD and ViCLIP Score) with a 5x smaller model (1.1B vs 5.2B). Finally, we show how VideoFlexTok can enable long video generation without prohibitive computational cost by training a text-to-video model on 10-second 81-frame videos with only 672 tokens, 8x fewer than a comparable 3D grid tokenizer.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-atanov26a, title = {{V}ideo{F}lex{T}ok: Flexible-Length Coarse-to-Fine Video Tokenization}, author = {Atanov, Andrei and Allardice, Jesse and Bachmann, Roman and Kar, O\u{g}uzhan Fatih and Hjelm, R Devon and Griffiths, David and Fu, Peter and Dehghan, Afshin and Zamir, Amir}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {4259--4282}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/atanov26a/atanov26a.pdf}, url = {https://proceedings.mlr.press/v306/atanov26a.html}, abstract = {Visual tokenizers map high-dimensional raw pixels into a compressed representation for downstream modeling. Beyond compression, tokenizers dictate what information is preserved and how it is organized. A de facto standard approach to video tokenization is to represent a video as a spatiotemporal 3D grid of tokens, each capturing local information from the original signal. This requires the downstream model, e.g., a text-to-video model, to learn to predict all low-level details "pixel-by-pixel" irrespective of the video’s inherent complexity, leading to high learning complexity. We present VideoFlexTok, which represents videos with a variable-length sequence of tokens structured in a coarse-to-fine manner, where the first tokens (emergently) capture information such as semantics and motion, and later tokens add fine-grained details. The generative flow decoder enables realistic video reconstructions from any token count. This representation structure allows adapting the token count to downstream needs and encoding videos longer than the baselines within the same budget. We evaluate VideoFlexTok on class- and text-to-video generative tasks and show that it yields more efficient training than 3D grid tokens, achieving comparable generation quality (gFVD and ViCLIP Score) with a 5x smaller model (1.1B vs 5.2B). Finally, we show how VideoFlexTok can enable long video generation without prohibitive computational cost by training a text-to-video model on 10-second 81-frame videos with only 672 tokens, 8x fewer than a comparable 3D grid tokenizer.} }
Endnote
%0 Conference Paper %T VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization %A Andrei Atanov %A Jesse Allardice %A Roman Bachmann %A Oğuzhan Fatih Kar %A R Devon Hjelm %A David Griffiths %A Peter Fu %A Afshin Dehghan %A Amir Zamir %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-atanov26a %I PMLR %P 4259--4282 %U https://proceedings.mlr.press/v306/atanov26a.html %V 306 %X Visual tokenizers map high-dimensional raw pixels into a compressed representation for downstream modeling. Beyond compression, tokenizers dictate what information is preserved and how it is organized. A de facto standard approach to video tokenization is to represent a video as a spatiotemporal 3D grid of tokens, each capturing local information from the original signal. This requires the downstream model, e.g., a text-to-video model, to learn to predict all low-level details "pixel-by-pixel" irrespective of the video’s inherent complexity, leading to high learning complexity. We present VideoFlexTok, which represents videos with a variable-length sequence of tokens structured in a coarse-to-fine manner, where the first tokens (emergently) capture information such as semantics and motion, and later tokens add fine-grained details. The generative flow decoder enables realistic video reconstructions from any token count. This representation structure allows adapting the token count to downstream needs and encoding videos longer than the baselines within the same budget. We evaluate VideoFlexTok on class- and text-to-video generative tasks and show that it yields more efficient training than 3D grid tokens, achieving comparable generation quality (gFVD and ViCLIP Score) with a 5x smaller model (1.1B vs 5.2B). Finally, we show how VideoFlexTok can enable long video generation without prohibitive computational cost by training a text-to-video model on 10-second 81-frame videos with only 672 tokens, 8x fewer than a comparable 3D grid tokenizer.
APA
Atanov, A., Allardice, J., Bachmann, R., Kar, O.F., Hjelm, R.D., Griffiths, D., Fu, P., Dehghan, A. & Zamir, A.. (2026). VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:4259-4282 Available from https://proceedings.mlr.press/v306/atanov26a.html.

Related Material