Spectrum Extraction and Clipping for Implicitly Linear Layers

Ali Ebrahimpour Boroojeny, Matus Telgarsky, Hari Sundaram
Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, PMLR 238:2971-2979, 2024.

Abstract

We show the effectiveness of automatic differentiation in efficiently and correctly computing and controlling the spectrum of implicitly linear operators, a rich family of layer types including all standard convolutional and dense layers. We provide the first clipping method which is correct for general convolution layers, and illuminate the representational limitation that caused correctness issues in prior work. We study the effect of the batch normalization layers when concatenated with convolutional layers and show how our clipping method can be applied to their composition. By comparing the accuracy and performance of our algorithms to the state-of-the-art methods, using various experiments, we show they are more precise and efficient and lead to better generalization and adversarial robustness. We provide the code for using our methods at https://github.com/Ali-E/FastClip.

Cite this Paper

BibTeX
@InProceedings{pmlr-v238-ebrahimpour-boroojeny24a, title = {Spectrum Extraction and Clipping for Implicitly Linear Layers}, author = {Ebrahimpour Boroojeny, Ali and Telgarsky, Matus and Sundaram, Hari}, booktitle = {Proceedings of The 27th International Conference on Artificial Intelligence and Statistics}, pages = {2971--2979}, year = {2024}, editor = {Dasgupta, Sanjoy and Mandt, Stephan and Li, Yingzhen}, volume = {238}, series = {Proceedings of Machine Learning Research}, month = {02--04 May}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v238/ebrahimpour-boroojeny24a/ebrahimpour-boroojeny24a.pdf}, url = {https://proceedings.mlr.press/v238/ebrahimpour-boroojeny24a.html}, abstract = {We show the effectiveness of automatic differentiation in efficiently and correctly computing and controlling the spectrum of implicitly linear operators, a rich family of layer types including all standard convolutional and dense layers. We provide the first clipping method which is correct for general convolution layers, and illuminate the representational limitation that caused correctness issues in prior work. We study the effect of the batch normalization layers when concatenated with convolutional layers and show how our clipping method can be applied to their composition. By comparing the accuracy and performance of our algorithms to the state-of-the-art methods, using various experiments, we show they are more precise and efficient and lead to better generalization and adversarial robustness. We provide the code for using our methods at https://github.com/Ali-E/FastClip.} }
Endnote
%0 Conference Paper %T Spectrum Extraction and Clipping for Implicitly Linear Layers %A Ali Ebrahimpour Boroojeny %A Matus Telgarsky %A Hari Sundaram %B Proceedings of The 27th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2024 %E Sanjoy Dasgupta %E Stephan Mandt %E Yingzhen Li %F pmlr-v238-ebrahimpour-boroojeny24a %I PMLR %P 2971--2979 %U https://proceedings.mlr.press/v238/ebrahimpour-boroojeny24a.html %V 238 %X We show the effectiveness of automatic differentiation in efficiently and correctly computing and controlling the spectrum of implicitly linear operators, a rich family of layer types including all standard convolutional and dense layers. We provide the first clipping method which is correct for general convolution layers, and illuminate the representational limitation that caused correctness issues in prior work. We study the effect of the batch normalization layers when concatenated with convolutional layers and show how our clipping method can be applied to their composition. By comparing the accuracy and performance of our algorithms to the state-of-the-art methods, using various experiments, we show they are more precise and efficient and lead to better generalization and adversarial robustness. We provide the code for using our methods at https://github.com/Ali-E/FastClip.
APA
Ebrahimpour Boroojeny, A., Telgarsky, M. & Sundaram, H.. (2024). Spectrum Extraction and Clipping for Implicitly Linear Layers. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 238:2971-2979 Available from https://proceedings.mlr.press/v238/ebrahimpour-boroojeny24a.html.

Related Material