Coarse-To-Fine Tensor Trains for Compact Visual Representations

Sebastian Bugge Loeschcke, Dan Wang, Christian Munklinde Leth-Espensen, Serge Belongie, Michael Kastoryano, Sagie Benaim
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:32612-32642, 2024.

Abstract

The ability to learn compact, high-quality, and easy-to-optimize representations for visual data is paramount to many applications such as novel view synthesis and 3D reconstruction. Recent work has shown substantial success in using tensor networks to design such compact and high-quality representations. However, the ability to optimize tensor-based representations, and in particular, the highly compact tensor train representation, is still lacking. This has prevented practitioners from deploying the full potential of tensor networks for visual data. To this end, we propose ’Prolongation Upsampling Tensor Train (PuTT)’, a novel method for learning tensor train representations in a coarse-to-fine manner. Our method involves the prolonging or ‘upsampling’ of a learned tensor train representation, creating a sequence of ’coarse-to-fine’ tensor trains that are incrementally refined. We evaluate our representation along three axes: (1). compression, (2). denoising capability, and (3). image completion capability. To assess these axes, we consider the tasks of image fitting, 3D fitting, and novel view synthesis, where our method shows an improved performance compared to state-of-the-art tensor-based methods.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-loeschcke24a, title = {Coarse-To-Fine Tensor Trains for Compact Visual Representations}, author = {Loeschcke, Sebastian Bugge and Wang, Dan and Leth-Espensen, Christian Munklinde and Belongie, Serge and Kastoryano, Michael and Benaim, Sagie}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {32612--32642}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/loeschcke24a/loeschcke24a.pdf}, url = {https://proceedings.mlr.press/v235/loeschcke24a.html}, abstract = {The ability to learn compact, high-quality, and easy-to-optimize representations for visual data is paramount to many applications such as novel view synthesis and 3D reconstruction. Recent work has shown substantial success in using tensor networks to design such compact and high-quality representations. However, the ability to optimize tensor-based representations, and in particular, the highly compact tensor train representation, is still lacking. This has prevented practitioners from deploying the full potential of tensor networks for visual data. To this end, we propose ’Prolongation Upsampling Tensor Train (PuTT)’, a novel method for learning tensor train representations in a coarse-to-fine manner. Our method involves the prolonging or ‘upsampling’ of a learned tensor train representation, creating a sequence of ’coarse-to-fine’ tensor trains that are incrementally refined. We evaluate our representation along three axes: (1). compression, (2). denoising capability, and (3). image completion capability. To assess these axes, we consider the tasks of image fitting, 3D fitting, and novel view synthesis, where our method shows an improved performance compared to state-of-the-art tensor-based methods.} }
Endnote
%0 Conference Paper %T Coarse-To-Fine Tensor Trains for Compact Visual Representations %A Sebastian Bugge Loeschcke %A Dan Wang %A Christian Munklinde Leth-Espensen %A Serge Belongie %A Michael Kastoryano %A Sagie Benaim %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-loeschcke24a %I PMLR %P 32612--32642 %U https://proceedings.mlr.press/v235/loeschcke24a.html %V 235 %X The ability to learn compact, high-quality, and easy-to-optimize representations for visual data is paramount to many applications such as novel view synthesis and 3D reconstruction. Recent work has shown substantial success in using tensor networks to design such compact and high-quality representations. However, the ability to optimize tensor-based representations, and in particular, the highly compact tensor train representation, is still lacking. This has prevented practitioners from deploying the full potential of tensor networks for visual data. To this end, we propose ’Prolongation Upsampling Tensor Train (PuTT)’, a novel method for learning tensor train representations in a coarse-to-fine manner. Our method involves the prolonging or ‘upsampling’ of a learned tensor train representation, creating a sequence of ’coarse-to-fine’ tensor trains that are incrementally refined. We evaluate our representation along three axes: (1). compression, (2). denoising capability, and (3). image completion capability. To assess these axes, we consider the tasks of image fitting, 3D fitting, and novel view synthesis, where our method shows an improved performance compared to state-of-the-art tensor-based methods.
APA
Loeschcke, S.B., Wang, D., Leth-Espensen, C.M., Belongie, S., Kastoryano, M. & Benaim, S.. (2024). Coarse-To-Fine Tensor Trains for Compact Visual Representations. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:32612-32642 Available from https://proceedings.mlr.press/v235/loeschcke24a.html.

Related Material