Adam Dziedzic,
John Paparrizos,
Sanjay Krishnan,
Aaron Elmore,
Michael Franklin
;
Proceedings of the 36th International Conference on Machine Learning, PMLR 97:1745-1754, 2019.
Abstract
The convolutional layers are core building blocks of neural network architectures. In general, a convolutional filter applies to the entire frequency spectrum of the input data. We explore artificially constraining the frequency spectra of these filters and data, called band-limiting, during training. The frequency domain constraints apply to both the feed-forward and back-propagation steps. Experimentally, we observe that Convolutional Neural Networks (CNNs) are resilient to this compression scheme and results suggest that CNNs learn to leverage lower-frequency components. In particular, we found: (1) band-limited training can effectively control the resource usage (GPU and memory); (2) models trained with band-limited layers retain high prediction accuracy; and (3) requires no modification to existing training algorithms or neural network architectures to use unlike other compression schemes.
@InProceedings{pmlr-v97-dziedzic19a,
title = {Band-limited Training and Inference for Convolutional Neural Networks},
author = {Dziedzic, Adam and Paparrizos, John and Krishnan, Sanjay and Elmore, Aaron and Franklin, Michael},
booktitle = {Proceedings of the 36th International Conference on Machine Learning},
pages = {1745--1754},
year = {2019},
editor = {Chaudhuri, Kamalika and Salakhutdinov, Ruslan},
volume = {97},
series = {Proceedings of Machine Learning Research},
address = {Long Beach, California, USA},
month = {09--15 Jun},
publisher = {PMLR},
pdf = {http://proceedings.mlr.press/v97/dziedzic19a/dziedzic19a.pdf},
url = {http://proceedings.mlr.press/v97/dziedzic19a.html},
abstract = {The convolutional layers are core building blocks of neural network architectures. In general, a convolutional filter applies to the entire frequency spectrum of the input data. We explore artificially constraining the frequency spectra of these filters and data, called band-limiting, during training. The frequency domain constraints apply to both the feed-forward and back-propagation steps. Experimentally, we observe that Convolutional Neural Networks (CNNs) are resilient to this compression scheme and results suggest that CNNs learn to leverage lower-frequency components. In particular, we found: (1) band-limited training can effectively control the resource usage (GPU and memory); (2) models trained with band-limited layers retain high prediction accuracy; and (3) requires no modification to existing training algorithms or neural network architectures to use unlike other compression schemes.}
}
%0 Conference Paper
%T Band-limited Training and Inference for Convolutional Neural Networks
%A Adam Dziedzic
%A John Paparrizos
%A Sanjay Krishnan
%A Aaron Elmore
%A Michael Franklin
%B Proceedings of the 36th International Conference on Machine Learning
%C Proceedings of Machine Learning Research
%D 2019
%E Kamalika Chaudhuri
%E Ruslan Salakhutdinov
%F pmlr-v97-dziedzic19a
%I PMLR
%J Proceedings of Machine Learning Research
%P 1745--1754
%U http://proceedings.mlr.press
%V 97
%W PMLR
%X The convolutional layers are core building blocks of neural network architectures. In general, a convolutional filter applies to the entire frequency spectrum of the input data. We explore artificially constraining the frequency spectra of these filters and data, called band-limiting, during training. The frequency domain constraints apply to both the feed-forward and back-propagation steps. Experimentally, we observe that Convolutional Neural Networks (CNNs) are resilient to this compression scheme and results suggest that CNNs learn to leverage lower-frequency components. In particular, we found: (1) band-limited training can effectively control the resource usage (GPU and memory); (2) models trained with band-limited layers retain high prediction accuracy; and (3) requires no modification to existing training algorithms or neural network architectures to use unlike other compression schemes.
Dziedzic, A., Paparrizos, J., Krishnan, S., Elmore, A. & Franklin, M.. (2019). Band-limited Training and Inference for Convolutional Neural Networks. Proceedings of the 36th International Conference on Machine Learning, in PMLR 97:1745-1754
This site last compiled Mon, 16 Sep 2019 16:05:04 +0000