Sparsity-Inducing Categorical Prior Improves Robustness of the Information Bottleneck

Anirban Samaddar, Sandeep Madireddy, Prasanna Balaprakash, Taps Maiti, Gustavo de los Campos, Ian Fischer
Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, PMLR 206:10207-10222, 2023.

Abstract

The information bottleneck framework provides a systematic approach to learning representations that compress nuisance information in the input and extract semantically meaningful information about predictions. However, the choice of a prior distribution that fixes the dimensionality across all the data can restrict the flexibility of this approach for learning robust representations. We present a novel sparsity-inducing spike-slab categorical prior that uses sparsity as a mechanism to provide the flexibility that allows each data point to learn its own dimension distribution. In addition, it provides a mechanism for learning a joint distribution of the latent variable and the sparsity, and hence it can account for the complete uncertainty in the latent space. Through a series of experiments using in-distribution and out-of-distribution learning scenarios on the MNIST, CIFAR-10, and ImageNet data, we show that the proposed approach improves accuracy and robustness compared to traditional fixed-dimensional priors, as well as other sparsity induction mechanisms for latent variable models proposed in the literature.

Cite this Paper

BibTeX
@InProceedings{pmlr-v206-samaddar23a, title = {Sparsity-Inducing Categorical Prior Improves Robustness of the Information Bottleneck}, author = {Samaddar, Anirban and Madireddy, Sandeep and Balaprakash, Prasanna and Maiti, Taps and de los Campos, Gustavo and Fischer, Ian}, booktitle = {Proceedings of The 26th International Conference on Artificial Intelligence and Statistics}, pages = {10207--10222}, year = {2023}, editor = {Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem}, volume = {206}, series = {Proceedings of Machine Learning Research}, month = {25--27 Apr}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v206/samaddar23a/samaddar23a.pdf}, url = {https://proceedings.mlr.press/v206/samaddar23a.html}, abstract = {The information bottleneck framework provides a systematic approach to learning representations that compress nuisance information in the input and extract semantically meaningful information about predictions. However, the choice of a prior distribution that fixes the dimensionality across all the data can restrict the flexibility of this approach for learning robust representations. We present a novel sparsity-inducing spike-slab categorical prior that uses sparsity as a mechanism to provide the flexibility that allows each data point to learn its own dimension distribution. In addition, it provides a mechanism for learning a joint distribution of the latent variable and the sparsity, and hence it can account for the complete uncertainty in the latent space. Through a series of experiments using in-distribution and out-of-distribution learning scenarios on the MNIST, CIFAR-10, and ImageNet data, we show that the proposed approach improves accuracy and robustness compared to traditional fixed-dimensional priors, as well as other sparsity induction mechanisms for latent variable models proposed in the literature.} }
Endnote
%0 Conference Paper %T Sparsity-Inducing Categorical Prior Improves Robustness of the Information Bottleneck %A Anirban Samaddar %A Sandeep Madireddy %A Prasanna Balaprakash %A Taps Maiti %A Gustavo de los Campos %A Ian Fischer %B Proceedings of The 26th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2023 %E Francisco Ruiz %E Jennifer Dy %E Jan-Willem van de Meent %F pmlr-v206-samaddar23a %I PMLR %P 10207--10222 %U https://proceedings.mlr.press/v206/samaddar23a.html %V 206 %X The information bottleneck framework provides a systematic approach to learning representations that compress nuisance information in the input and extract semantically meaningful information about predictions. However, the choice of a prior distribution that fixes the dimensionality across all the data can restrict the flexibility of this approach for learning robust representations. We present a novel sparsity-inducing spike-slab categorical prior that uses sparsity as a mechanism to provide the flexibility that allows each data point to learn its own dimension distribution. In addition, it provides a mechanism for learning a joint distribution of the latent variable and the sparsity, and hence it can account for the complete uncertainty in the latent space. Through a series of experiments using in-distribution and out-of-distribution learning scenarios on the MNIST, CIFAR-10, and ImageNet data, we show that the proposed approach improves accuracy and robustness compared to traditional fixed-dimensional priors, as well as other sparsity induction mechanisms for latent variable models proposed in the literature.
APA
Samaddar, A., Madireddy, S., Balaprakash, P., Maiti, T., de los Campos, G. & Fischer, I.. (2023). Sparsity-Inducing Categorical Prior Improves Robustness of the Information Bottleneck. Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 206:10207-10222 Available from https://proceedings.mlr.press/v206/samaddar23a.html.

Related Material