WSNet: Compact and Efficient Networks Through Weight Sampling

Xiaojie Jin, Yingzhen Yang, Ning Xu, Jianchao Yang, Nebojsa Jojic, Jiashi Feng, Shuicheng Yan
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:2352-2361, 2018.

Abstract

We present a new approach and a novel architecture, termed WSNet, for learning compact and efficient deep neural networks. Existing approaches conventionally learn full model parameters independently and then compress them via ad hoc processing such as model pruning or filter factorization. Alternatively, WSNet proposes learning model parameters by sampling from a compact set of learnable parameters, which naturally enforces parameter sharing throughout the learning process. We demonstrate that such a novel weight sampling approach (and induced WSNet) promotes both weights and computation sharing favorably. By employing this method, we can more efficiently learn much smaller networks with competitive performance compared to baseline networks with equal numbers of convolution filters. Specifically, we consider learning compact and efficient 1D convolutional neural networks for audio classification. Extensive experiments on multiple audio classification datasets verify the effectiveness of WSNet. Combined with weight quantization, the resulted models are up to 180x smaller and theoretically up to 16x faster than the well-established baselines, without noticeable performance drop.

Cite this Paper

BibTeX
@InProceedings{pmlr-v80-jin18d, title = {{WSN}et: Compact and Efficient Networks Through Weight Sampling}, author = {Jin, Xiaojie and Yang, Yingzhen and Xu, Ning and Yang, Jianchao and Jojic, Nebojsa and Feng, Jiashi and Yan, Shuicheng}, booktitle = {Proceedings of the 35th International Conference on Machine Learning}, pages = {2352--2361}, year = {2018}, editor = {Dy, Jennifer and Krause, Andreas}, volume = {80}, series = {Proceedings of Machine Learning Research}, month = {10--15 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v80/jin18d/jin18d.pdf}, url = {https://proceedings.mlr.press/v80/jin18d.html}, abstract = {We present a new approach and a novel architecture, termed WSNet, for learning compact and efficient deep neural networks. Existing approaches conventionally learn full model parameters independently and then compress them via ad hoc processing such as model pruning or filter factorization. Alternatively, WSNet proposes learning model parameters by sampling from a compact set of learnable parameters, which naturally enforces parameter sharing throughout the learning process. We demonstrate that such a novel weight sampling approach (and induced WSNet) promotes both weights and computation sharing favorably. By employing this method, we can more efficiently learn much smaller networks with competitive performance compared to baseline networks with equal numbers of convolution filters. Specifically, we consider learning compact and efficient 1D convolutional neural networks for audio classification. Extensive experiments on multiple audio classification datasets verify the effectiveness of WSNet. Combined with weight quantization, the resulted models are up to 180x smaller and theoretically up to 16x faster than the well-established baselines, without noticeable performance drop.} }
Endnote
%0 Conference Paper %T WSNet: Compact and Efficient Networks Through Weight Sampling %A Xiaojie Jin %A Yingzhen Yang %A Ning Xu %A Jianchao Yang %A Nebojsa Jojic %A Jiashi Feng %A Shuicheng Yan %B Proceedings of the 35th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2018 %E Jennifer Dy %E Andreas Krause %F pmlr-v80-jin18d %I PMLR %P 2352--2361 %U https://proceedings.mlr.press/v80/jin18d.html %V 80 %X We present a new approach and a novel architecture, termed WSNet, for learning compact and efficient deep neural networks. Existing approaches conventionally learn full model parameters independently and then compress them via ad hoc processing such as model pruning or filter factorization. Alternatively, WSNet proposes learning model parameters by sampling from a compact set of learnable parameters, which naturally enforces parameter sharing throughout the learning process. We demonstrate that such a novel weight sampling approach (and induced WSNet) promotes both weights and computation sharing favorably. By employing this method, we can more efficiently learn much smaller networks with competitive performance compared to baseline networks with equal numbers of convolution filters. Specifically, we consider learning compact and efficient 1D convolutional neural networks for audio classification. Extensive experiments on multiple audio classification datasets verify the effectiveness of WSNet. Combined with weight quantization, the resulted models are up to 180x smaller and theoretically up to 16x faster than the well-established baselines, without noticeable performance drop.
APA
Jin, X., Yang, Y., Xu, N., Yang, J., Jojic, N., Feng, J. & Yan, S.. (2018). WSNet: Compact and Efficient Networks Through Weight Sampling. Proceedings of the 35th International Conference on Machine Learning, in Proceedings of Machine Learning Research 80:2352-2361 Available from https://proceedings.mlr.press/v80/jin18d.html.

Related Material