Progressive Cramming: Reliable Token Compression and What It Reveals

Dmitrii Tarasov, Timofei Lashukov, Elizaveta Goncharova, Andrey Kuznetsov
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:119780-119815, 2026.

Abstract

Token cramming compresses sequences into learned embeddings with near-perfect reconstruction, but fixed token budgets and 99% accuracy thresholds leave it unclear whether residual errors reflect optimization failures or fundamental limits. We introduce progressive cramming, which grows the target prefix token-by-token, stopping only when reconstruction is no longer achievable within a fixed optimization budget. Progressive trajectories occupy low-dimensional structure in embedding space. Prepending a crammed embedding causes a moderate but consistent accuracy drop on multiple-choice benchmarks even with the original prefix in context, and collapses capability almost entirely under generative evaluation. Causal attention-knockout interventions trace this degradation to the embedding’s interactions in the model’s early layers. These results position progressive cramming as a tool for studying compression limits and show that perfect reconstruction - achievable through brittle steering rather than transferable semantics - is insufficient for meaningful compression.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-tarasov26a, title = {Progressive Cramming: Reliable Token Compression and What It Reveals}, author = {Tarasov, Dmitrii and Lashukov, Timofei and Goncharova, Elizaveta and Kuznetsov, Andrey}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {119780--119815}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/tarasov26a/tarasov26a.pdf}, url = {https://proceedings.mlr.press/v306/tarasov26a.html}, abstract = {Token cramming compresses sequences into learned embeddings with near-perfect reconstruction, but fixed token budgets and 99% accuracy thresholds leave it unclear whether residual errors reflect optimization failures or fundamental limits. We introduce progressive cramming, which grows the target prefix token-by-token, stopping only when reconstruction is no longer achievable within a fixed optimization budget. Progressive trajectories occupy low-dimensional structure in embedding space. Prepending a crammed embedding causes a moderate but consistent accuracy drop on multiple-choice benchmarks even with the original prefix in context, and collapses capability almost entirely under generative evaluation. Causal attention-knockout interventions trace this degradation to the embedding’s interactions in the model’s early layers. These results position progressive cramming as a tool for studying compression limits and show that perfect reconstruction - achievable through brittle steering rather than transferable semantics - is insufficient for meaningful compression.} }
Endnote
%0 Conference Paper %T Progressive Cramming: Reliable Token Compression and What It Reveals %A Dmitrii Tarasov %A Timofei Lashukov %A Elizaveta Goncharova %A Andrey Kuznetsov %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-tarasov26a %I PMLR %P 119780--119815 %U https://proceedings.mlr.press/v306/tarasov26a.html %V 306 %X Token cramming compresses sequences into learned embeddings with near-perfect reconstruction, but fixed token budgets and 99% accuracy thresholds leave it unclear whether residual errors reflect optimization failures or fundamental limits. We introduce progressive cramming, which grows the target prefix token-by-token, stopping only when reconstruction is no longer achievable within a fixed optimization budget. Progressive trajectories occupy low-dimensional structure in embedding space. Prepending a crammed embedding causes a moderate but consistent accuracy drop on multiple-choice benchmarks even with the original prefix in context, and collapses capability almost entirely under generative evaluation. Causal attention-knockout interventions trace this degradation to the embedding’s interactions in the model’s early layers. These results position progressive cramming as a tool for studying compression limits and show that perfect reconstruction - achievable through brittle steering rather than transferable semantics - is insufficient for meaningful compression.
APA
Tarasov, D., Lashukov, T., Goncharova, E. & Kuznetsov, A.. (2026). Progressive Cramming: Reliable Token Compression and What It Reveals. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:119780-119815 Available from https://proceedings.mlr.press/v306/tarasov26a.html.

Related Material