Learning Class-relevant Features and Class-irrelevant Features via a Hybrid third-order RBM

Heng Luo, Ruimin Shen, Changyong Niu, Carsten Ullrich
Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, PMLR 15:470-478, 2011.

Abstract

Restricted Boltzmann Machines are commonly used in unsupervised learning to extract features from training data. Since these features are learned for regenerating training data a classifier based on them has to be trained. If only a few of the learned features are discriminative other non-discriminative features will distract the classifier during the training process and thus waste computing resources for testing. In this paper, we present a hybrid third-order Restricted Boltzmann Machine in which class-relevant features (for recognizing) and class-irrelevant features (for generating only) are learned simultaneously. As the classification task uses only the class-relevant features, the test itself becomes very fast. We show that class-irrelevant features help class-relevant features to focus on the recognition task and introduce useful regularization effects to reduce the norms of class-relevant features. Thus there is no need to use weight-decay for the parameters of this model. Experiments on the MNIST, NORB and Caltech101 Silhouettes datasets show very promising results.

Cite this Paper

BibTeX
@InProceedings{pmlr-v15-luo11a, title = {Learning Class-relevant Features and Class-irrelevant Features via a Hybrid third-order RBM}, author = {Luo, Heng and Shen, Ruimin and Niu, Changyong and Ullrich, Carsten}, booktitle = {Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics}, pages = {470--478}, year = {2011}, editor = {Gordon, Geoffrey and Dunson, David and Dudík, Miroslav}, volume = {15}, series = {Proceedings of Machine Learning Research}, address = {Fort Lauderdale, FL, USA}, month = {11--13 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v15/luo11a/luo11a.pdf}, url = {https://proceedings.mlr.press/v15/luo11a.html}, abstract = {Restricted Boltzmann Machines are commonly used in unsupervised learning to extract features from training data. Since these features are learned for regenerating training data a classifier based on them has to be trained. If only a few of the learned features are discriminative other non-discriminative features will distract the classifier during the training process and thus waste computing resources for testing. In this paper, we present a hybrid third-order Restricted Boltzmann Machine in which class-relevant features (for recognizing) and class-irrelevant features (for generating only) are learned simultaneously. As the classification task uses only the class-relevant features, the test itself becomes very fast. We show that class-irrelevant features help class-relevant features to focus on the recognition task and introduce useful regularization effects to reduce the norms of class-relevant features. Thus there is no need to use weight-decay for the parameters of this model. Experiments on the MNIST, NORB and Caltech101 Silhouettes datasets show very promising results.} }
Endnote
%0 Conference Paper %T Learning Class-relevant Features and Class-irrelevant Features via a Hybrid third-order RBM %A Heng Luo %A Ruimin Shen %A Changyong Niu %A Carsten Ullrich %B Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2011 %E Geoffrey Gordon %E David Dunson %E Miroslav Dudík %F pmlr-v15-luo11a %I PMLR %P 470--478 %U https://proceedings.mlr.press/v15/luo11a.html %V 15 %X Restricted Boltzmann Machines are commonly used in unsupervised learning to extract features from training data. Since these features are learned for regenerating training data a classifier based on them has to be trained. If only a few of the learned features are discriminative other non-discriminative features will distract the classifier during the training process and thus waste computing resources for testing. In this paper, we present a hybrid third-order Restricted Boltzmann Machine in which class-relevant features (for recognizing) and class-irrelevant features (for generating only) are learned simultaneously. As the classification task uses only the class-relevant features, the test itself becomes very fast. We show that class-irrelevant features help class-relevant features to focus on the recognition task and introduce useful regularization effects to reduce the norms of class-relevant features. Thus there is no need to use weight-decay for the parameters of this model. Experiments on the MNIST, NORB and Caltech101 Silhouettes datasets show very promising results.
RIS
TY - CPAPER TI - Learning Class-relevant Features and Class-irrelevant Features via a Hybrid third-order RBM AU - Heng Luo AU - Ruimin Shen AU - Changyong Niu AU - Carsten Ullrich BT - Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics DA - 2011/06/14 ED - Geoffrey Gordon ED - David Dunson ED - Miroslav Dudík ID - pmlr-v15-luo11a PB - PMLR DP - Proceedings of Machine Learning Research VL - 15 SP - 470 EP - 478 L1 - http://proceedings.mlr.press/v15/luo11a/luo11a.pdf UR - https://proceedings.mlr.press/v15/luo11a.html AB - Restricted Boltzmann Machines are commonly used in unsupervised learning to extract features from training data. Since these features are learned for regenerating training data a classifier based on them has to be trained. If only a few of the learned features are discriminative other non-discriminative features will distract the classifier during the training process and thus waste computing resources for testing. In this paper, we present a hybrid third-order Restricted Boltzmann Machine in which class-relevant features (for recognizing) and class-irrelevant features (for generating only) are learned simultaneously. As the classification task uses only the class-relevant features, the test itself becomes very fast. We show that class-irrelevant features help class-relevant features to focus on the recognition task and introduce useful regularization effects to reduce the norms of class-relevant features. Thus there is no need to use weight-decay for the parameters of this model. Experiments on the MNIST, NORB and Caltech101 Silhouettes datasets show very promising results. ER -
APA
Luo, H., Shen, R., Niu, C. & Ullrich, C.. (2011). Learning Class-relevant Features and Class-irrelevant Features via a Hybrid third-order RBM. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 15:470-478 Available from https://proceedings.mlr.press/v15/luo11a.html.

Related Material