Equivariance Discovery by Learned Parameter-Sharing

Raymond A. Yeh, Yuan-Ting Hu, Mark Hasegawa-Johnson, Alexander Schwing
Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, PMLR 151:1527-1545, 2022.

Abstract

Designing equivariance as an inductive bias into deep-nets has been a prominent approach to build effective models, e.g., a convolutional neural network incorporates translation equivariance. However, incorporating these inductive biases requires knowledge about the equivariance properties of the data, which may not be available, e.g., when encountering a new domain. To address this, we study how to "discover interpretable equivariances" from data. Specifically, we formulate this discovery process as an optimization problem over a model’s parameter-sharing schemes. We propose to use the partition distance to empirically quantify the accuracy of the recovered equivariance. Also, we theoretically analyze the method for Gaussian data and provide a bound on the mean squared gap between the studied discovery scheme and the oracle scheme. Empirically, we show that the approach recovers known equivariances, such as permutations and shifts, on sum of numbers and spatially-invariant data.

Cite this Paper

BibTeX
@InProceedings{pmlr-v151-yeh22b, title = { Equivariance Discovery by Learned Parameter-Sharing }, author = {Yeh, Raymond A. and Hu, Yuan-Ting and Hasegawa-Johnson, Mark and Schwing, Alexander}, booktitle = {Proceedings of The 25th International Conference on Artificial Intelligence and Statistics}, pages = {1527--1545}, year = {2022}, editor = {Camps-Valls, Gustau and Ruiz, Francisco J. R. and Valera, Isabel}, volume = {151}, series = {Proceedings of Machine Learning Research}, month = {28--30 Mar}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v151/yeh22b/yeh22b.pdf}, url = {https://proceedings.mlr.press/v151/yeh22b.html}, abstract = { Designing equivariance as an inductive bias into deep-nets has been a prominent approach to build effective models, e.g., a convolutional neural network incorporates translation equivariance. However, incorporating these inductive biases requires knowledge about the equivariance properties of the data, which may not be available, e.g., when encountering a new domain. To address this, we study how to "discover interpretable equivariances" from data. Specifically, we formulate this discovery process as an optimization problem over a model’s parameter-sharing schemes. We propose to use the partition distance to empirically quantify the accuracy of the recovered equivariance. Also, we theoretically analyze the method for Gaussian data and provide a bound on the mean squared gap between the studied discovery scheme and the oracle scheme. Empirically, we show that the approach recovers known equivariances, such as permutations and shifts, on sum of numbers and spatially-invariant data. } }
Endnote
%0 Conference Paper %T Equivariance Discovery by Learned Parameter-Sharing %A Raymond A. Yeh %A Yuan-Ting Hu %A Mark Hasegawa-Johnson %A Alexander Schwing %B Proceedings of The 25th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2022 %E Gustau Camps-Valls %E Francisco J. R. Ruiz %E Isabel Valera %F pmlr-v151-yeh22b %I PMLR %P 1527--1545 %U https://proceedings.mlr.press/v151/yeh22b.html %V 151 %X Designing equivariance as an inductive bias into deep-nets has been a prominent approach to build effective models, e.g., a convolutional neural network incorporates translation equivariance. However, incorporating these inductive biases requires knowledge about the equivariance properties of the data, which may not be available, e.g., when encountering a new domain. To address this, we study how to "discover interpretable equivariances" from data. Specifically, we formulate this discovery process as an optimization problem over a model’s parameter-sharing schemes. We propose to use the partition distance to empirically quantify the accuracy of the recovered equivariance. Also, we theoretically analyze the method for Gaussian data and provide a bound on the mean squared gap between the studied discovery scheme and the oracle scheme. Empirically, we show that the approach recovers known equivariances, such as permutations and shifts, on sum of numbers and spatially-invariant data.
APA
Yeh, R.A., Hu, Y., Hasegawa-Johnson, M. & Schwing, A.. (2022). Equivariance Discovery by Learned Parameter-Sharing . Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 151:1527-1545 Available from https://proceedings.mlr.press/v151/yeh22b.html.

Related Material