Learning Discrete Representations via Information Maximizing Self-Augmented Training

Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, Masashi Sugiyama
Proceedings of the 34th International Conference on Machine Learning, PMLR 70:1558-1567, 2017.

Abstract

Learning discrete representations of data is a central machine learning task because of the compactness of the representations and ease of interpretation. The task includes clustering and hash learning as special cases. Deep neural networks are promising to be used because they can model the non-linearity of data and scale to large datasets. However, their model complexity is huge, and therefore, we need to carefully regularize the networks in order to learn useful representations that exhibit intended invariance for applications of interest. To this end, we propose a method called Information Maximizing Self-Augmented Training (IMSAT). In IMSAT, we use data augmentation to impose the invariance on discrete representations. More specifically, we encourage the predicted representations of augmented data points to be close to those of the original data points in an end-to-end fashion. At the same time, we maximize the information-theoretic dependency between data and their predicted discrete representations. Extensive experiments on benchmark datasets show that IMSAT produces state-of-the-art results for both clustering and unsupervised hash learning.

Cite this Paper

BibTeX
@InProceedings{pmlr-v70-hu17b, title = {Learning Discrete Representations via Information Maximizing Self-Augmented Training}, author = {Weihua Hu and Takeru Miyato and Seiya Tokui and Eiichi Matsumoto and Masashi Sugiyama}, booktitle = {Proceedings of the 34th International Conference on Machine Learning}, pages = {1558--1567}, year = {2017}, editor = {Precup, Doina and Teh, Yee Whye}, volume = {70}, series = {Proceedings of Machine Learning Research}, month = {06--11 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v70/hu17b/hu17b.pdf}, url = {https://proceedings.mlr.press/v70/hu17b.html}, abstract = {Learning discrete representations of data is a central machine learning task because of the compactness of the representations and ease of interpretation. The task includes clustering and hash learning as special cases. Deep neural networks are promising to be used because they can model the non-linearity of data and scale to large datasets. However, their model complexity is huge, and therefore, we need to carefully regularize the networks in order to learn useful representations that exhibit intended invariance for applications of interest. To this end, we propose a method called Information Maximizing Self-Augmented Training (IMSAT). In IMSAT, we use data augmentation to impose the invariance on discrete representations. More specifically, we encourage the predicted representations of augmented data points to be close to those of the original data points in an end-to-end fashion. At the same time, we maximize the information-theoretic dependency between data and their predicted discrete representations. Extensive experiments on benchmark datasets show that IMSAT produces state-of-the-art results for both clustering and unsupervised hash learning.} }
Endnote
%0 Conference Paper %T Learning Discrete Representations via Information Maximizing Self-Augmented Training %A Weihua Hu %A Takeru Miyato %A Seiya Tokui %A Eiichi Matsumoto %A Masashi Sugiyama %B Proceedings of the 34th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2017 %E Doina Precup %E Yee Whye Teh %F pmlr-v70-hu17b %I PMLR %P 1558--1567 %U https://proceedings.mlr.press/v70/hu17b.html %V 70 %X Learning discrete representations of data is a central machine learning task because of the compactness of the representations and ease of interpretation. The task includes clustering and hash learning as special cases. Deep neural networks are promising to be used because they can model the non-linearity of data and scale to large datasets. However, their model complexity is huge, and therefore, we need to carefully regularize the networks in order to learn useful representations that exhibit intended invariance for applications of interest. To this end, we propose a method called Information Maximizing Self-Augmented Training (IMSAT). In IMSAT, we use data augmentation to impose the invariance on discrete representations. More specifically, we encourage the predicted representations of augmented data points to be close to those of the original data points in an end-to-end fashion. At the same time, we maximize the information-theoretic dependency between data and their predicted discrete representations. Extensive experiments on benchmark datasets show that IMSAT produces state-of-the-art results for both clustering and unsupervised hash learning.
APA
Hu, W., Miyato, T., Tokui, S., Matsumoto, E. & Sugiyama, M.. (2017). Learning Discrete Representations via Information Maximizing Self-Augmented Training. Proceedings of the 34th International Conference on Machine Learning, in Proceedings of Machine Learning Research 70:1558-1567 Available from https://proceedings.mlr.press/v70/hu17b.html.

Related Material