Training Binary Neural Networks through Learning with Noisy Supervision

Kai Han, Yunhe Wang, Yixing Xu, Chunjing Xu, Enhua Wu, Chang Xu
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:4017-4026, 2020.

Abstract

This paper formalizes the binarization operations over neural networks from a learning perspective. In contrast to classical hand crafted rules (\eg hard thresholding) to binarize full-precision neurons, we propose to learn a mapping from full-precision neurons to the target binary ones. Each individual weight entry will not be binarized independently. Instead, they are taken as a whole to accomplish the binarization, just as they work together in generating convolution features. To help the training of the binarization mapping, the full-precision neurons after taking sign operations is regarded as some auxiliary supervision signal, which is noisy but still has valuable guidance. An unbiased estimator is therefore introduced to mitigate the influence of the supervision noise. Experimental results on benchmark datasets indicate that the proposed binarization technique attains consistent improvements over baselines.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-han20d, title = {Training Binary Neural Networks through Learning with Noisy Supervision}, author = {Han, Kai and Wang, Yunhe and Xu, Yixing and Xu, Chunjing and Wu, Enhua and Xu, Chang}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {4017--4026}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/han20d/han20d.pdf}, url = {https://proceedings.mlr.press/v119/han20d.html}, abstract = {This paper formalizes the binarization operations over neural networks from a learning perspective. In contrast to classical hand crafted rules (\eg hard thresholding) to binarize full-precision neurons, we propose to learn a mapping from full-precision neurons to the target binary ones. Each individual weight entry will not be binarized independently. Instead, they are taken as a whole to accomplish the binarization, just as they work together in generating convolution features. To help the training of the binarization mapping, the full-precision neurons after taking sign operations is regarded as some auxiliary supervision signal, which is noisy but still has valuable guidance. An unbiased estimator is therefore introduced to mitigate the influence of the supervision noise. Experimental results on benchmark datasets indicate that the proposed binarization technique attains consistent improvements over baselines.} }
Endnote
%0 Conference Paper %T Training Binary Neural Networks through Learning with Noisy Supervision %A Kai Han %A Yunhe Wang %A Yixing Xu %A Chunjing Xu %A Enhua Wu %A Chang Xu %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-han20d %I PMLR %P 4017--4026 %U https://proceedings.mlr.press/v119/han20d.html %V 119 %X This paper formalizes the binarization operations over neural networks from a learning perspective. In contrast to classical hand crafted rules (\eg hard thresholding) to binarize full-precision neurons, we propose to learn a mapping from full-precision neurons to the target binary ones. Each individual weight entry will not be binarized independently. Instead, they are taken as a whole to accomplish the binarization, just as they work together in generating convolution features. To help the training of the binarization mapping, the full-precision neurons after taking sign operations is regarded as some auxiliary supervision signal, which is noisy but still has valuable guidance. An unbiased estimator is therefore introduced to mitigate the influence of the supervision noise. Experimental results on benchmark datasets indicate that the proposed binarization technique attains consistent improvements over baselines.
APA
Han, K., Wang, Y., Xu, Y., Xu, C., Wu, E. & Xu, C.. (2020). Training Binary Neural Networks through Learning with Noisy Supervision. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:4017-4026 Available from https://proceedings.mlr.press/v119/han20d.html.

Related Material