Provably Learning Diverse Features in Multi-View Data with Midpoint Mixup

Muthu Chidambaram, Xiang Wang, Chenwei Wu, Rong Ge
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:5563-5599, 2023.

Abstract

Mixup is a data augmentation technique that relies on training using random convex combinations of data points and their labels. In recent years, Mixup has become a standard primitive used in the training of state-of-the-art image classification models due to its demonstrated benefits over empirical risk minimization with regards to generalization and robustness. In this work, we try to explain some of this success from a feature learning perspective. We focus our attention on classification problems in which each class may have multiple associated features (or $\textit{views}$) that can be used to predict the class correctly. Our main theoretical results demonstrate that, for a non-trivial class of data distributions with two features per class, training a 2-layer convolutional network using empirical risk minimization can lead to learning only one feature for almost all classes while training with a specific instantiation of Mixup succeeds in learning both features for every class. We also show empirically that these theoretical insights extend to the practical settings of image benchmarks modified to have multiple features.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-chidambaram23a, title = {Provably Learning Diverse Features in Multi-View Data with Midpoint Mixup}, author = {Chidambaram, Muthu and Wang, Xiang and Wu, Chenwei and Ge, Rong}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {5563--5599}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/chidambaram23a/chidambaram23a.pdf}, url = {https://proceedings.mlr.press/v202/chidambaram23a.html}, abstract = {Mixup is a data augmentation technique that relies on training using random convex combinations of data points and their labels. In recent years, Mixup has become a standard primitive used in the training of state-of-the-art image classification models due to its demonstrated benefits over empirical risk minimization with regards to generalization and robustness. In this work, we try to explain some of this success from a feature learning perspective. We focus our attention on classification problems in which each class may have multiple associated features (or $\textit{views}$) that can be used to predict the class correctly. Our main theoretical results demonstrate that, for a non-trivial class of data distributions with two features per class, training a 2-layer convolutional network using empirical risk minimization can lead to learning only one feature for almost all classes while training with a specific instantiation of Mixup succeeds in learning both features for every class. We also show empirically that these theoretical insights extend to the practical settings of image benchmarks modified to have multiple features.} }
Endnote
%0 Conference Paper %T Provably Learning Diverse Features in Multi-View Data with Midpoint Mixup %A Muthu Chidambaram %A Xiang Wang %A Chenwei Wu %A Rong Ge %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-chidambaram23a %I PMLR %P 5563--5599 %U https://proceedings.mlr.press/v202/chidambaram23a.html %V 202 %X Mixup is a data augmentation technique that relies on training using random convex combinations of data points and their labels. In recent years, Mixup has become a standard primitive used in the training of state-of-the-art image classification models due to its demonstrated benefits over empirical risk minimization with regards to generalization and robustness. In this work, we try to explain some of this success from a feature learning perspective. We focus our attention on classification problems in which each class may have multiple associated features (or $\textit{views}$) that can be used to predict the class correctly. Our main theoretical results demonstrate that, for a non-trivial class of data distributions with two features per class, training a 2-layer convolutional network using empirical risk minimization can lead to learning only one feature for almost all classes while training with a specific instantiation of Mixup succeeds in learning both features for every class. We also show empirically that these theoretical insights extend to the practical settings of image benchmarks modified to have multiple features.
APA
Chidambaram, M., Wang, X., Wu, C. & Ge, R.. (2023). Provably Learning Diverse Features in Multi-View Data with Midpoint Mixup. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:5563-5599 Available from https://proceedings.mlr.press/v202/chidambaram23a.html.

Related Material