SparseProp: Efficient Sparse Backpropagation for Faster Training of Neural Networks at the Edge

Mahdi Nikdan, Tommaso Pegolotti, Eugenia Iofinova, Eldar Kurtic, Dan Alistarh
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:26215-26227, 2023.

Abstract

We provide an efficient implementation of the backpropagation algorithm, specialized to the case where the weights of the neural network being trained are sparse. Our algorithm is general, as it applies to arbitrary (unstructured) sparsity and common layer types (e.g., convolutional or linear). We provide a fast vectorized implementation on commodity CPUs, and show that it can yield speedups in end-to-end runtime experiments, both in transfer learning using already-sparsified networks, and in training sparse networks from scratch. Thus, our results provide the first support for sparse training on commodity hardware.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-nikdan23a, title = {{S}parse{P}rop: Efficient Sparse Backpropagation for Faster Training of Neural Networks at the Edge}, author = {Nikdan, Mahdi and Pegolotti, Tommaso and Iofinova, Eugenia and Kurtic, Eldar and Alistarh, Dan}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {26215--26227}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/nikdan23a/nikdan23a.pdf}, url = {https://proceedings.mlr.press/v202/nikdan23a.html}, abstract = {We provide an efficient implementation of the backpropagation algorithm, specialized to the case where the weights of the neural network being trained are sparse. Our algorithm is general, as it applies to arbitrary (unstructured) sparsity and common layer types (e.g., convolutional or linear). We provide a fast vectorized implementation on commodity CPUs, and show that it can yield speedups in end-to-end runtime experiments, both in transfer learning using already-sparsified networks, and in training sparse networks from scratch. Thus, our results provide the first support for sparse training on commodity hardware.} }
Endnote
%0 Conference Paper %T SparseProp: Efficient Sparse Backpropagation for Faster Training of Neural Networks at the Edge %A Mahdi Nikdan %A Tommaso Pegolotti %A Eugenia Iofinova %A Eldar Kurtic %A Dan Alistarh %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-nikdan23a %I PMLR %P 26215--26227 %U https://proceedings.mlr.press/v202/nikdan23a.html %V 202 %X We provide an efficient implementation of the backpropagation algorithm, specialized to the case where the weights of the neural network being trained are sparse. Our algorithm is general, as it applies to arbitrary (unstructured) sparsity and common layer types (e.g., convolutional or linear). We provide a fast vectorized implementation on commodity CPUs, and show that it can yield speedups in end-to-end runtime experiments, both in transfer learning using already-sparsified networks, and in training sparse networks from scratch. Thus, our results provide the first support for sparse training on commodity hardware.
APA
Nikdan, M., Pegolotti, T., Iofinova, E., Kurtic, E. & Alistarh, D.. (2023). SparseProp: Efficient Sparse Backpropagation for Faster Training of Neural Networks at the Edge. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:26215-26227 Available from https://proceedings.mlr.press/v202/nikdan23a.html.

Related Material