Band-limited Training and Inference for Convolutional Neural Networks

Adam Dziedzic; John Paparrizos; Sanjay Krishnan; Aaron Elmore; Michael Franklin

Band-limited Training and Inference for Convolutional Neural Networks

Adam Dziedzic, John Paparrizos, Sanjay Krishnan, Aaron Elmore, Michael Franklin ;

Proceedings of the 36th International Conference on Machine Learning, PMLR 97:1745-1754, 2019.

Abstract

The convolutional layers are core building blocks of neural network architectures. In general, a convolutional filter applies to the entire frequency spectrum of the input data. We explore artificially constraining the frequency spectra of these filters and data, called band-limiting, during training. The frequency domain constraints apply to both the feed-forward and back-propagation steps. Experimentally, we observe that Convolutional Neural Networks (CNNs) are resilient to this compression scheme and results suggest that CNNs learn to leverage lower-frequency components. In particular, we found: (1) band-limited training can effectively control the resource usage (GPU and memory); (2) models trained with band-limited layers retain high prediction accuracy; and (3) requires no modification to existing training algorithms or neural network architectures to use unlike other compression schemes.

Related Material

@InProceedings{pmlr-v97-dziedzic19a, title = {Band-limited Training and Inference for Convolutional Neural Networks}, author = {Dziedzic, Adam and Paparrizos, John and Krishnan, Sanjay and Elmore, Aaron and Franklin, Michael}, booktitle = {Proceedings of the 36th International Conference on Machine Learning}, pages = {1745--1754}, year = {2019}, editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan}, volume = {97}, series = {Proceedings of Machine Learning Research}, address = {Long Beach, California, USA}, month = {09--15 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v97/dziedzic19a/dziedzic19a.pdf}, url = {http://proceedings.mlr.press/v97/dziedzic19a.html}, abstract = {The convolutional layers are core building blocks of neural network architectures. In general, a convolutional filter applies to the entire frequency spectrum of the input data. We explore artificially constraining the frequency spectra of these filters and data, called band-limiting, during training. The frequency domain constraints apply to both the feed-forward and back-propagation steps. Experimentally, we observe that Convolutional Neural Networks (CNNs) are resilient to this compression scheme and results suggest that CNNs learn to leverage lower-frequency components. In particular, we found: (1) band-limited training can effectively control the resource usage (GPU and memory); (2) models trained with band-limited layers retain high prediction accuracy; and (3) requires no modification to existing training algorithms or neural network architectures to use unlike other compression schemes.} }

%0 Conference Paper %T Band-limited Training and Inference for Convolutional Neural Networks %A Adam Dziedzic %A John Paparrizos %A Sanjay Krishnan %A Aaron Elmore %A Michael Franklin %B Proceedings of the 36th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Ruslan Salakhutdinov %F pmlr-v97-dziedzic19a %I PMLR %J Proceedings of Machine Learning Research %P 1745--1754 %U http://proceedings.mlr.press %V 97 %W PMLR %X The convolutional layers are core building blocks of neural network architectures. In general, a convolutional filter applies to the entire frequency spectrum of the input data. We explore artificially constraining the frequency spectra of these filters and data, called band-limiting, during training. The frequency domain constraints apply to both the feed-forward and back-propagation steps. Experimentally, we observe that Convolutional Neural Networks (CNNs) are resilient to this compression scheme and results suggest that CNNs learn to leverage lower-frequency components. In particular, we found: (1) band-limited training can effectively control the resource usage (GPU and memory); (2) models trained with band-limited layers retain high prediction accuracy; and (3) requires no modification to existing training algorithms or neural network architectures to use unlike other compression schemes.

Dziedzic, A., Paparrizos, J., Krishnan, S., Elmore, A. & Franklin, M.. (2019). Band-limited Training and Inference for Convolutional Neural Networks. Proceedings of the 36th International Conference on Machine Learning, in PMLR 97:1745-1754