Data Augmentation as Feature Manipulation

Ruoqi Shen, Sebastien Bubeck, Suriya Gunasekar
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:19773-19808, 2022.

Abstract

Data augmentation is a cornerstone of the machine learning pipeline, yet its theoretical underpinnings remain unclear. Is it merely a way to artificially augment the data set size? Or is it about encouraging the model to satisfy certain invariances? In this work we consider another angle, and we study the effect of data augmentation on the dynamic of the learning process. We find that data augmentation can alter the relative importance of various features, effectively making certain informative but hard to learn features more likely to be captured in the learning process. Importantly, we show that this effect is more pronounced for non-linear models, such as neural networks. Our main contribution is a detailed analysis of data augmentation on the learning dynamic for a two layer convolutional neural network in the recently proposed multi-view model by Z. Allen-Zhu and Y. Li. We complement this analysis with further experimental evidence that data augmentation can be viewed as a form of feature manipulation.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-shen22a, title = {Data Augmentation as Feature Manipulation}, author = {Shen, Ruoqi and Bubeck, Sebastien and Gunasekar, Suriya}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {19773--19808}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/shen22a/shen22a.pdf}, url = {https://proceedings.mlr.press/v162/shen22a.html}, abstract = {Data augmentation is a cornerstone of the machine learning pipeline, yet its theoretical underpinnings remain unclear. Is it merely a way to artificially augment the data set size? Or is it about encouraging the model to satisfy certain invariances? In this work we consider another angle, and we study the effect of data augmentation on the dynamic of the learning process. We find that data augmentation can alter the relative importance of various features, effectively making certain informative but hard to learn features more likely to be captured in the learning process. Importantly, we show that this effect is more pronounced for non-linear models, such as neural networks. Our main contribution is a detailed analysis of data augmentation on the learning dynamic for a two layer convolutional neural network in the recently proposed multi-view model by Z. Allen-Zhu and Y. Li. We complement this analysis with further experimental evidence that data augmentation can be viewed as a form of feature manipulation.} }
Endnote
%0 Conference Paper %T Data Augmentation as Feature Manipulation %A Ruoqi Shen %A Sebastien Bubeck %A Suriya Gunasekar %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-shen22a %I PMLR %P 19773--19808 %U https://proceedings.mlr.press/v162/shen22a.html %V 162 %X Data augmentation is a cornerstone of the machine learning pipeline, yet its theoretical underpinnings remain unclear. Is it merely a way to artificially augment the data set size? Or is it about encouraging the model to satisfy certain invariances? In this work we consider another angle, and we study the effect of data augmentation on the dynamic of the learning process. We find that data augmentation can alter the relative importance of various features, effectively making certain informative but hard to learn features more likely to be captured in the learning process. Importantly, we show that this effect is more pronounced for non-linear models, such as neural networks. Our main contribution is a detailed analysis of data augmentation on the learning dynamic for a two layer convolutional neural network in the recently proposed multi-view model by Z. Allen-Zhu and Y. Li. We complement this analysis with further experimental evidence that data augmentation can be viewed as a form of feature manipulation.
APA
Shen, R., Bubeck, S. & Gunasekar, S.. (2022). Data Augmentation as Feature Manipulation. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:19773-19808 Available from https://proceedings.mlr.press/v162/shen22a.html.

Related Material