Operation-Aware Soft Channel Pruning using Differentiable Masks

Minsoo Kang, Bohyung Han
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:5122-5131, 2020.

Abstract

We propose a simple but effective data-driven channel pruning algorithm, which compresses deep neural networks in a differentiable way by exploiting the characteristics of operations. The proposed approach makes a joint consideration of batch normalization (BN) and rectified linear unit (ReLU) for channel pruning; it estimates how likely the two successive operations deactivate each feature map and prunes the channels with high probabilities. To this end, we learn differentiable masks for individual channels and make soft decisions throughout the optimization procedure, which facilitates to explore larger search space and train more stable networks. The proposed framework enables us to identify compressed models via a joint learning of model parameters and channel pruning without an extra procedure of fine-tuning. We perform extensive experiments and achieve outstanding performance in terms of the accuracy of output networks given the same amount of resources when compared with the state-of-the-art methods.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-kang20a, title = {Operation-Aware Soft Channel Pruning using Differentiable Masks}, author = {Kang, Minsoo and Han, Bohyung}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {5122--5131}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/kang20a/kang20a.pdf}, url = {https://proceedings.mlr.press/v119/kang20a.html}, abstract = {We propose a simple but effective data-driven channel pruning algorithm, which compresses deep neural networks in a differentiable way by exploiting the characteristics of operations. The proposed approach makes a joint consideration of batch normalization (BN) and rectified linear unit (ReLU) for channel pruning; it estimates how likely the two successive operations deactivate each feature map and prunes the channels with high probabilities. To this end, we learn differentiable masks for individual channels and make soft decisions throughout the optimization procedure, which facilitates to explore larger search space and train more stable networks. The proposed framework enables us to identify compressed models via a joint learning of model parameters and channel pruning without an extra procedure of fine-tuning. We perform extensive experiments and achieve outstanding performance in terms of the accuracy of output networks given the same amount of resources when compared with the state-of-the-art methods.} }
Endnote
%0 Conference Paper %T Operation-Aware Soft Channel Pruning using Differentiable Masks %A Minsoo Kang %A Bohyung Han %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-kang20a %I PMLR %P 5122--5131 %U https://proceedings.mlr.press/v119/kang20a.html %V 119 %X We propose a simple but effective data-driven channel pruning algorithm, which compresses deep neural networks in a differentiable way by exploiting the characteristics of operations. The proposed approach makes a joint consideration of batch normalization (BN) and rectified linear unit (ReLU) for channel pruning; it estimates how likely the two successive operations deactivate each feature map and prunes the channels with high probabilities. To this end, we learn differentiable masks for individual channels and make soft decisions throughout the optimization procedure, which facilitates to explore larger search space and train more stable networks. The proposed framework enables us to identify compressed models via a joint learning of model parameters and channel pruning without an extra procedure of fine-tuning. We perform extensive experiments and achieve outstanding performance in terms of the accuracy of output networks given the same amount of resources when compared with the state-of-the-art methods.
APA
Kang, M. & Han, B.. (2020). Operation-Aware Soft Channel Pruning using Differentiable Masks. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:5122-5131 Available from https://proceedings.mlr.press/v119/kang20a.html.

Related Material