Disentanglement by Prediction: A Twin Autoencoder with a Factored Latent Traveler

Tegan Emerson, Audun D Myers, Sarah McGuire Scullen, Tim Marrinan
Proceedings of the 2nd Conference on Topology, Algebra, and Geometry in Data Science(TAG-DS 2026), PMLR 334(2):333-344, 2026.

Abstract

We recast disentangled representation learning as latent-space prediction with a factored predictor. Existing approaches—$\beta$-VAE, FactorVAE, $\beta$-TCVAE—cast disentanglement as a statistical property of a marginal latent distribution: they identify what factors describe an image but cannot predict how the latent should move when a factor changes. Drawing on the joint-embedding predictive architecture (JEPA) framework, in which predicting in latent space yields structured representations, we ask: what is the right structure for that predictor when the world is governed by independent generative factors? We propose the Factored-Traversal Twin Autoencoder (FT-TAE): a JEPA-inspired twin-encoder model in which a discrete factor predictor (Gumbel-Softmax) identifies what changed between two views and a learnable state-embedding table $\mathbf{E}\!\in\!\mathbb{R}^{K\times S\times d}$ composes the latent transition as a sum of per-factor displacements. Each displacement is the difference between two table lookups, so unchanged factors contribute exactly zero by construction and the additive form is order-independent without any explicit regularizer. The system is trained end-to-end with a single reconstruction loss—disentanglement emerges as a byproduct of learning accurate factored transitions, not from a marginal-statistics objective. We evaluate on 3D Shapes and MPI3D, measuring factor alignment, traversal accuracy, and compositionality.

Cite this Paper

BibTeX
@InProceedings{pmlr-v334-emerson26a, title = {Disentanglement by Prediction: A Twin Autoencoder with a Factored Latent Traveler}, author = {Emerson, Tegan and Myers, Audun D and Scullen, Sarah McGuire and Marrinan, Tim}, booktitle = {Proceedings of the 2nd Conference on Topology, Algebra, and Geometry in Data Science(TAG-DS 2026)}, pages = {333--344}, year = {2026}, editor = {Berman, Eddie and Bernárdez, Guillermo and Chen, Samantha and Cloninger, Alex and Doster, Timothy and Emerson, Tegan and Grigsby, J. Elisenda and Kvinge, Henry and Lawrence, Hannah and Marrinan, Tim and Myers, Audun and Papillon, Mathilde and Tahmasebi, Behrooz and Telyatnikov, Lev and Walters, Robin and Weber, Melanie and Xie, YuQing and Yeats, Eric}, volume = {334}, number = {2}, series = {Proceedings of Machine Learning Research}, month = {18--20 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v334/main/assets/emerson26a/emerson26a.pdf}, url = {https://proceedings.mlr.press/v334/emerson26a.html}, abstract = {We recast disentangled representation learning as latent-space prediction with a factored predictor. Existing approaches—$\beta$-VAE, FactorVAE, $\beta$-TCVAE—cast disentanglement as a statistical property of a marginal latent distribution: they identify what factors describe an image but cannot predict how the latent should move when a factor changes. Drawing on the joint-embedding predictive architecture (JEPA) framework, in which predicting in latent space yields structured representations, we ask: what is the right structure for that predictor when the world is governed by independent generative factors? We propose the Factored-Traversal Twin Autoencoder (FT-TAE): a JEPA-inspired twin-encoder model in which a discrete factor predictor (Gumbel-Softmax) identifies what changed between two views and a learnable state-embedding table $\mathbf{E}\!\in\!\mathbb{R}^{K\times S\times d}$ composes the latent transition as a sum of per-factor displacements. Each displacement is the difference between two table lookups, so unchanged factors contribute exactly zero by construction and the additive form is order-independent without any explicit regularizer. The system is trained end-to-end with a single reconstruction loss—disentanglement emerges as a byproduct of learning accurate factored transitions, not from a marginal-statistics objective. We evaluate on 3D Shapes and MPI3D, measuring factor alignment, traversal accuracy, and compositionality.} }
Endnote
%0 Conference Paper %T Disentanglement by Prediction: A Twin Autoencoder with a Factored Latent Traveler %A Tegan Emerson %A Audun D Myers %A Sarah McGuire Scullen %A Tim Marrinan %B Proceedings of the 2nd Conference on Topology, Algebra, and Geometry in Data Science(TAG-DS 2026) %C Proceedings of Machine Learning Research %D 2026 %E Eddie Berman %E Guillermo Bernárdez %E Samantha Chen %E Alex Cloninger %E Timothy Doster %E Tegan Emerson %E J. Elisenda Grigsby %E Henry Kvinge %E Hannah Lawrence %E Tim Marrinan %E Audun Myers %E Mathilde Papillon %E Behrooz Tahmasebi %E Lev Telyatnikov %E Robin Walters %E Melanie Weber %E YuQing Xie %E Eric Yeats %F pmlr-v334-emerson26a %I PMLR %P 333--344 %U https://proceedings.mlr.press/v334/emerson26a.html %V 334 %N 2 %X We recast disentangled representation learning as latent-space prediction with a factored predictor. Existing approaches—$\beta$-VAE, FactorVAE, $\beta$-TCVAE—cast disentanglement as a statistical property of a marginal latent distribution: they identify what factors describe an image but cannot predict how the latent should move when a factor changes. Drawing on the joint-embedding predictive architecture (JEPA) framework, in which predicting in latent space yields structured representations, we ask: what is the right structure for that predictor when the world is governed by independent generative factors? We propose the Factored-Traversal Twin Autoencoder (FT-TAE): a JEPA-inspired twin-encoder model in which a discrete factor predictor (Gumbel-Softmax) identifies what changed between two views and a learnable state-embedding table $\mathbf{E}\!\in\!\mathbb{R}^{K\times S\times d}$ composes the latent transition as a sum of per-factor displacements. Each displacement is the difference between two table lookups, so unchanged factors contribute exactly zero by construction and the additive form is order-independent without any explicit regularizer. The system is trained end-to-end with a single reconstruction loss—disentanglement emerges as a byproduct of learning accurate factored transitions, not from a marginal-statistics objective. We evaluate on 3D Shapes and MPI3D, measuring factor alignment, traversal accuracy, and compositionality.
APA
Emerson, T., Myers, A.D., Scullen, S.M. & Marrinan, T.. (2026). Disentanglement by Prediction: A Twin Autoencoder with a Factored Latent Traveler. Proceedings of the 2nd Conference on Topology, Algebra, and Geometry in Data Science(TAG-DS 2026), in Proceedings of Machine Learning Research 334(2):333-344 Available from https://proceedings.mlr.press/v334/emerson26a.html.

Related Material