Indirectly Parameterized Concrete Autoencoders

Alfred Nilsson, Klas Wijk, Sai Bharath Chandra Gutha, Erik Englesson, Alexandra Hotti, Carlo Saccardi, Oskar Kviman, Jens Lagergren, Ricardo Vinuesa Motilva, Hossein Azizpour
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:38237-38252, 2024.

Abstract

Feature selection is a crucial task in settings where data is high-dimensional or acquiring the full set of features is costly. Recent developments in neural network-based embedded feature selection show promising results across a wide range of applications. Concrete Autoencoders (CAEs), considered state-of-the-art in embedded feature selection, may struggle to achieve stable joint optimization, hurting their training time and generalization. In this work, we identify that this instability is correlated with the CAE learning duplicate selections. To remedy this, we propose a simple and effective improvement: Indirectly Parameterized CAEs (IP-CAEs). IP-CAEs learn an embedding and a mapping from it to the Gumbel-Softmax distributions’ parameters. Despite being simple to implement, IP-CAE exhibits significant and consistent improvements over CAE in both generalization and training time across several datasets for reconstruction and classification. Unlike CAE, IP-CAE effectively leverages non-linear relationships and does not require retraining the jointly optimized decoder. Furthermore, our approach is, in principle, generalizable to Gumbel-Softmax distributions beyond feature selection.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-nilsson24b, title = {Indirectly Parameterized Concrete Autoencoders}, author = {Nilsson, Alfred and Wijk, Klas and Gutha, Sai Bharath Chandra and Englesson, Erik and Hotti, Alexandra and Saccardi, Carlo and Kviman, Oskar and Lagergren, Jens and Motilva, Ricardo Vinuesa and Azizpour, Hossein}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {38237--38252}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/nilsson24b/nilsson24b.pdf}, url = {https://proceedings.mlr.press/v235/nilsson24b.html}, abstract = {Feature selection is a crucial task in settings where data is high-dimensional or acquiring the full set of features is costly. Recent developments in neural network-based embedded feature selection show promising results across a wide range of applications. Concrete Autoencoders (CAEs), considered state-of-the-art in embedded feature selection, may struggle to achieve stable joint optimization, hurting their training time and generalization. In this work, we identify that this instability is correlated with the CAE learning duplicate selections. To remedy this, we propose a simple and effective improvement: Indirectly Parameterized CAEs (IP-CAEs). IP-CAEs learn an embedding and a mapping from it to the Gumbel-Softmax distributions’ parameters. Despite being simple to implement, IP-CAE exhibits significant and consistent improvements over CAE in both generalization and training time across several datasets for reconstruction and classification. Unlike CAE, IP-CAE effectively leverages non-linear relationships and does not require retraining the jointly optimized decoder. Furthermore, our approach is, in principle, generalizable to Gumbel-Softmax distributions beyond feature selection.} }
Endnote
%0 Conference Paper %T Indirectly Parameterized Concrete Autoencoders %A Alfred Nilsson %A Klas Wijk %A Sai Bharath Chandra Gutha %A Erik Englesson %A Alexandra Hotti %A Carlo Saccardi %A Oskar Kviman %A Jens Lagergren %A Ricardo Vinuesa Motilva %A Hossein Azizpour %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-nilsson24b %I PMLR %P 38237--38252 %U https://proceedings.mlr.press/v235/nilsson24b.html %V 235 %X Feature selection is a crucial task in settings where data is high-dimensional or acquiring the full set of features is costly. Recent developments in neural network-based embedded feature selection show promising results across a wide range of applications. Concrete Autoencoders (CAEs), considered state-of-the-art in embedded feature selection, may struggle to achieve stable joint optimization, hurting their training time and generalization. In this work, we identify that this instability is correlated with the CAE learning duplicate selections. To remedy this, we propose a simple and effective improvement: Indirectly Parameterized CAEs (IP-CAEs). IP-CAEs learn an embedding and a mapping from it to the Gumbel-Softmax distributions’ parameters. Despite being simple to implement, IP-CAE exhibits significant and consistent improvements over CAE in both generalization and training time across several datasets for reconstruction and classification. Unlike CAE, IP-CAE effectively leverages non-linear relationships and does not require retraining the jointly optimized decoder. Furthermore, our approach is, in principle, generalizable to Gumbel-Softmax distributions beyond feature selection.
APA
Nilsson, A., Wijk, K., Gutha, S.B.C., Englesson, E., Hotti, A., Saccardi, C., Kviman, O., Lagergren, J., Motilva, R.V. & Azizpour, H.. (2024). Indirectly Parameterized Concrete Autoencoders. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:38237-38252 Available from https://proceedings.mlr.press/v235/nilsson24b.html.

Related Material