Cell Variational Information Bottleneck Network

Zhonghua Zhai
Proceedings of the 15th Asian Conference on Machine Learning, PMLR 222:1606-1621, 2024.

Abstract

In this work, we propose “Cell Variational Information Bottleneck Network (cellVIB)”, a convolutional neural network using information bottleneck mechanism, which can be combined with the latest feedforward network architecture in an end-to-end training method. Our Cell Variational Information Bottleneck Network is constructed by stacking VIB cells, which generate feature maps with uncertainty. As layers going deeper, the regularization effect will gradually increase, instead of directly adding excessive regular constraints to the output layer of the model as in Deep VIB. In each VIB cell, the feedforward process learns an independent mean term and a standard deviation term, and predicts the Gaussian distribution based on them. The feedback process is based on reparameterization trick for effective training. This work performs an extensive analysis on MNIST dataset to verify the effectiveness of each VIB cells mentioned above, and provides an insightful analysis on how the VIB cells affect mutual information. Experiments conducted on CIFAR-10 also prove that our network is robust against noisy labels during training and against corrupted images during testing. Then, we validate our method on PACS dataset, whose results show that the VIB cells can significantly improve the generalization performance of the basic model. Finally, in a more complex representation learning task, face recognition, our network structure has also achieved very competitive results.

Cite this Paper

BibTeX
@InProceedings{pmlr-v222-zhai24a, title = {Cell Variational Information Bottleneck Network}, author = {Zhai, Zhonghua}, booktitle = {Proceedings of the 15th Asian Conference on Machine Learning}, pages = {1606--1621}, year = {2024}, editor = {Yanıkoğlu, Berrin and Buntine, Wray}, volume = {222}, series = {Proceedings of Machine Learning Research}, month = {11--14 Nov}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v222/zhai24a/zhai24a.pdf}, url = {https://proceedings.mlr.press/v222/zhai24a.html}, abstract = {In this work, we propose “Cell Variational Information Bottleneck Network (cellVIB)”, a convolutional neural network using information bottleneck mechanism, which can be combined with the latest feedforward network architecture in an end-to-end training method. Our Cell Variational Information Bottleneck Network is constructed by stacking VIB cells, which generate feature maps with uncertainty. As layers going deeper, the regularization effect will gradually increase, instead of directly adding excessive regular constraints to the output layer of the model as in Deep VIB. In each VIB cell, the feedforward process learns an independent mean term and a standard deviation term, and predicts the Gaussian distribution based on them. The feedback process is based on reparameterization trick for effective training. This work performs an extensive analysis on MNIST dataset to verify the effectiveness of each VIB cells mentioned above, and provides an insightful analysis on how the VIB cells affect mutual information. Experiments conducted on CIFAR-10 also prove that our network is robust against noisy labels during training and against corrupted images during testing. Then, we validate our method on PACS dataset, whose results show that the VIB cells can significantly improve the generalization performance of the basic model. Finally, in a more complex representation learning task, face recognition, our network structure has also achieved very competitive results.} }
Endnote
%0 Conference Paper %T Cell Variational Information Bottleneck Network %A Zhonghua Zhai %B Proceedings of the 15th Asian Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Berrin Yanıkoğlu %E Wray Buntine %F pmlr-v222-zhai24a %I PMLR %P 1606--1621 %U https://proceedings.mlr.press/v222/zhai24a.html %V 222 %X In this work, we propose “Cell Variational Information Bottleneck Network (cellVIB)”, a convolutional neural network using information bottleneck mechanism, which can be combined with the latest feedforward network architecture in an end-to-end training method. Our Cell Variational Information Bottleneck Network is constructed by stacking VIB cells, which generate feature maps with uncertainty. As layers going deeper, the regularization effect will gradually increase, instead of directly adding excessive regular constraints to the output layer of the model as in Deep VIB. In each VIB cell, the feedforward process learns an independent mean term and a standard deviation term, and predicts the Gaussian distribution based on them. The feedback process is based on reparameterization trick for effective training. This work performs an extensive analysis on MNIST dataset to verify the effectiveness of each VIB cells mentioned above, and provides an insightful analysis on how the VIB cells affect mutual information. Experiments conducted on CIFAR-10 also prove that our network is robust against noisy labels during training and against corrupted images during testing. Then, we validate our method on PACS dataset, whose results show that the VIB cells can significantly improve the generalization performance of the basic model. Finally, in a more complex representation learning task, face recognition, our network structure has also achieved very competitive results.
APA
Zhai, Z.. (2024). Cell Variational Information Bottleneck Network. Proceedings of the 15th Asian Conference on Machine Learning, in Proceedings of Machine Learning Research 222:1606-1621 Available from https://proceedings.mlr.press/v222/zhai24a.html.

Related Material