Developing Autoencoder: Incremental Bottleneck Expansion Leads to an Informed Latent Space

Deyue Kong, Jonas Elpelt, David Vogenauer, Markos Genios, Matthias Kaschube
Proceedings of the First Workshop on NeuroAI Multimodal Intelligence @ AAAI 2026, PMLR 308:58-66, 2026.

Abstract

Representation learning models, such as autoencoders (AEs), can effectively extract meaningful and generalizable features from natural image data. However, the learned latent features are often mixed or distributed across all bottleneck units, making interpretation difficult. Previous work has sought to address this by explicitly optimizing for feature separation or ordering. We propose a biologically inspired progressive learning scheme, the Developing Autoencoder (Dev-AE), which incrementally expands the representational capacity. Increasing the size of the bottleneck layer over training epochs forces the Dev-AE to first learn compressed, low-dimensional representations before expanding into progressively higher-dimensional feature spaces. Comparing the latent space organization in Dev-AEs with that in standard AEs and PCA-initialized AEs (PCA-AE), we observe improved feature ordering and higher activation sparsity. Moreover, Dev-AEs show better classification performance based on the learned encodings, with units added in the final increment contributing the most. Our findings indicate that an incremental latent space expansion fosters ordered, sparse, and more diverse representations, leading to more efficient use of representational capacity and improved classification accuracy, thereby offering a promising route toward interpretable and compact encodings.

Cite this Paper

BibTeX
@InProceedings{pmlr-v308-kong26a, title = {Developing Autoencoder: Incremental Bottleneck Expansion Leads to an Informed Latent Space}, author = {Kong, Deyue and Elpelt, Jonas and Vogenauer, David and Genios, Markos and Kaschube, Matthias}, booktitle = {Proceedings of the First Workshop on NeuroAI Multimodal Intelligence @ AAAI 2026}, pages = {58--66}, year = {2026}, editor = {Abbasi-Asl, Reza and Iqbal, Asim and Ito, Shinya and Arkhipov, Anton and Sanborn, Sophia}, volume = {308}, series = {Proceedings of Machine Learning Research}, month = {27 Jan}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v308/main/assets/kong26a/kong26a.pdf}, url = {https://proceedings.mlr.press/v308/kong26a.html}, abstract = {Representation learning models, such as autoencoders (AEs), can effectively extract meaningful and generalizable features from natural image data. However, the learned latent features are often mixed or distributed across all bottleneck units, making interpretation difficult. Previous work has sought to address this by explicitly optimizing for feature separation or ordering. We propose a biologically inspired progressive learning scheme, the Developing Autoencoder (Dev-AE), which incrementally expands the representational capacity. Increasing the size of the bottleneck layer over training epochs forces the Dev-AE to first learn compressed, low-dimensional representations before expanding into progressively higher-dimensional feature spaces. Comparing the latent space organization in Dev-AEs with that in standard AEs and PCA-initialized AEs (PCA-AE), we observe improved feature ordering and higher activation sparsity. Moreover, Dev-AEs show better classification performance based on the learned encodings, with units added in the final increment contributing the most. Our findings indicate that an incremental latent space expansion fosters ordered, sparse, and more diverse representations, leading to more efficient use of representational capacity and improved classification accuracy, thereby offering a promising route toward interpretable and compact encodings.} }
Endnote
%0 Conference Paper %T Developing Autoencoder: Incremental Bottleneck Expansion Leads to an Informed Latent Space %A Deyue Kong %A Jonas Elpelt %A David Vogenauer %A Markos Genios %A Matthias Kaschube %B Proceedings of the First Workshop on NeuroAI Multimodal Intelligence @ AAAI 2026 %C Proceedings of Machine Learning Research %D 2026 %E Reza Abbasi-Asl %E Asim Iqbal %E Shinya Ito %E Anton Arkhipov %E Sophia Sanborn %F pmlr-v308-kong26a %I PMLR %P 58--66 %U https://proceedings.mlr.press/v308/kong26a.html %V 308 %X Representation learning models, such as autoencoders (AEs), can effectively extract meaningful and generalizable features from natural image data. However, the learned latent features are often mixed or distributed across all bottleneck units, making interpretation difficult. Previous work has sought to address this by explicitly optimizing for feature separation or ordering. We propose a biologically inspired progressive learning scheme, the Developing Autoencoder (Dev-AE), which incrementally expands the representational capacity. Increasing the size of the bottleneck layer over training epochs forces the Dev-AE to first learn compressed, low-dimensional representations before expanding into progressively higher-dimensional feature spaces. Comparing the latent space organization in Dev-AEs with that in standard AEs and PCA-initialized AEs (PCA-AE), we observe improved feature ordering and higher activation sparsity. Moreover, Dev-AEs show better classification performance based on the learned encodings, with units added in the final increment contributing the most. Our findings indicate that an incremental latent space expansion fosters ordered, sparse, and more diverse representations, leading to more efficient use of representational capacity and improved classification accuracy, thereby offering a promising route toward interpretable and compact encodings.
APA
Kong, D., Elpelt, J., Vogenauer, D., Genios, M. & Kaschube, M.. (2026). Developing Autoencoder: Incremental Bottleneck Expansion Leads to an Informed Latent Space. Proceedings of the First Workshop on NeuroAI Multimodal Intelligence @ AAAI 2026, in Proceedings of Machine Learning Research 308:58-66 Available from https://proceedings.mlr.press/v308/kong26a.html.

Related Material