Squared-loss Mutual Information Regularization: A Novel Information-theoretic Approach to Semi-supervised Learning

Gang Niu, Wittawat Jitkrittum, Bo Dai, Hirotaka Hachiya, Masashi Sugiyama
Proceedings of the 30th International Conference on Machine Learning, PMLR 28(3):10-18, 2013.

Abstract

We propose squared-loss mutual information regularization (SMIR) for multi-class probabilistic classification, following the information maximization principle. SMIR is convex under mild conditions and thus improves the nonconvexity of mutual information regularization. It offers all of the following four abilities to semi-supervised algorithms: Analytical solution, out-of-sample/multi-class classification, and probabilistic output. Furthermore, novel generalization error bounds are derived. Experiments show SMIR compares favorably with state-of-the-art methods.

Cite this Paper

BibTeX
@InProceedings{pmlr-v28-niu13, title = {Squared-loss Mutual Information Regularization: A Novel Information-theoretic Approach to Semi-supervised Learning}, author = {Niu, Gang and Jitkrittum, Wittawat and Dai, Bo and Hachiya, Hirotaka and Sugiyama, Masashi}, booktitle = {Proceedings of the 30th International Conference on Machine Learning}, pages = {10--18}, year = {2013}, editor = {Dasgupta, Sanjoy and McAllester, David}, volume = {28}, number = {3}, series = {Proceedings of Machine Learning Research}, address = {Atlanta, Georgia, USA}, month = {17--19 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v28/niu13.pdf}, url = {https://proceedings.mlr.press/v28/niu13.html}, abstract = {We propose squared-loss mutual information regularization (SMIR) for multi-class probabilistic classification, following the information maximization principle. SMIR is convex under mild conditions and thus improves the nonconvexity of mutual information regularization. It offers all of the following four abilities to semi-supervised algorithms: Analytical solution, out-of-sample/multi-class classification, and probabilistic output. Furthermore, novel generalization error bounds are derived. Experiments show SMIR compares favorably with state-of-the-art methods.} }
Endnote
%0 Conference Paper %T Squared-loss Mutual Information Regularization: A Novel Information-theoretic Approach to Semi-supervised Learning %A Gang Niu %A Wittawat Jitkrittum %A Bo Dai %A Hirotaka Hachiya %A Masashi Sugiyama %B Proceedings of the 30th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2013 %E Sanjoy Dasgupta %E David McAllester %F pmlr-v28-niu13 %I PMLR %P 10--18 %U https://proceedings.mlr.press/v28/niu13.html %V 28 %N 3 %X We propose squared-loss mutual information regularization (SMIR) for multi-class probabilistic classification, following the information maximization principle. SMIR is convex under mild conditions and thus improves the nonconvexity of mutual information regularization. It offers all of the following four abilities to semi-supervised algorithms: Analytical solution, out-of-sample/multi-class classification, and probabilistic output. Furthermore, novel generalization error bounds are derived. Experiments show SMIR compares favorably with state-of-the-art methods.
RIS
TY - CPAPER TI - Squared-loss Mutual Information Regularization: A Novel Information-theoretic Approach to Semi-supervised Learning AU - Gang Niu AU - Wittawat Jitkrittum AU - Bo Dai AU - Hirotaka Hachiya AU - Masashi Sugiyama BT - Proceedings of the 30th International Conference on Machine Learning DA - 2013/05/26 ED - Sanjoy Dasgupta ED - David McAllester ID - pmlr-v28-niu13 PB - PMLR DP - Proceedings of Machine Learning Research VL - 28 IS - 3 SP - 10 EP - 18 L1 - http://proceedings.mlr.press/v28/niu13.pdf UR - https://proceedings.mlr.press/v28/niu13.html AB - We propose squared-loss mutual information regularization (SMIR) for multi-class probabilistic classification, following the information maximization principle. SMIR is convex under mild conditions and thus improves the nonconvexity of mutual information regularization. It offers all of the following four abilities to semi-supervised algorithms: Analytical solution, out-of-sample/multi-class classification, and probabilistic output. Furthermore, novel generalization error bounds are derived. Experiments show SMIR compares favorably with state-of-the-art methods. ER -
APA
Niu, G., Jitkrittum, W., Dai, B., Hachiya, H. & Sugiyama, M.. (2013). Squared-loss Mutual Information Regularization: A Novel Information-theoretic Approach to Semi-supervised Learning. Proceedings of the 30th International Conference on Machine Learning, in Proceedings of Machine Learning Research 28(3):10-18 Available from https://proceedings.mlr.press/v28/niu13.html.

Related Material