On Distributionally Robust Optimization and Data Rebalancing

Agnieszka Słowik, Leon Bottou
Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, PMLR 151:1283-1297, 2022.

Abstract

Machine learning systems based on minimizing average error have been shown to perform inconsistently across notable subsets of the data, which is not exposed by a low average error for the entire dataset. Distributionally Robust Optimization (DRO) seemingly addresses this problem by minimizing the worst expected risk across subpopulations. We establish theoretical results that clarify the relation between DRO and the optimization of the same loss averaged on an adequately weighted training dataset. The results cover finite and infinite number of training distributions, as well as convex and non-convex loss functions. An implication of our results is that for each DRO problem there exists a data distribution such that learning this distribution is equivalent to solving the DRO problem. Yet, important problems that DRO seeks to address (for instance, adversarial robustness and fighting bias) cannot be reduced to finding the one ’unbiased’ dataset. Our discussion section addresses this important discrepancy.

Cite this Paper

BibTeX
@InProceedings{pmlr-v151-slowik22a, title = { On Distributionally Robust Optimization and Data Rebalancing }, author = {S{\l}owik, Agnieszka and Bottou, Leon}, booktitle = {Proceedings of The 25th International Conference on Artificial Intelligence and Statistics}, pages = {1283--1297}, year = {2022}, editor = {Camps-Valls, Gustau and Ruiz, Francisco J. R. and Valera, Isabel}, volume = {151}, series = {Proceedings of Machine Learning Research}, month = {28--30 Mar}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v151/slowik22a/slowik22a.pdf}, url = {https://proceedings.mlr.press/v151/slowik22a.html}, abstract = { Machine learning systems based on minimizing average error have been shown to perform inconsistently across notable subsets of the data, which is not exposed by a low average error for the entire dataset. Distributionally Robust Optimization (DRO) seemingly addresses this problem by minimizing the worst expected risk across subpopulations. We establish theoretical results that clarify the relation between DRO and the optimization of the same loss averaged on an adequately weighted training dataset. The results cover finite and infinite number of training distributions, as well as convex and non-convex loss functions. An implication of our results is that for each DRO problem there exists a data distribution such that learning this distribution is equivalent to solving the DRO problem. Yet, important problems that DRO seeks to address (for instance, adversarial robustness and fighting bias) cannot be reduced to finding the one ’unbiased’ dataset. Our discussion section addresses this important discrepancy. } }
Endnote
%0 Conference Paper %T On Distributionally Robust Optimization and Data Rebalancing %A Agnieszka Słowik %A Leon Bottou %B Proceedings of The 25th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2022 %E Gustau Camps-Valls %E Francisco J. R. Ruiz %E Isabel Valera %F pmlr-v151-slowik22a %I PMLR %P 1283--1297 %U https://proceedings.mlr.press/v151/slowik22a.html %V 151 %X Machine learning systems based on minimizing average error have been shown to perform inconsistently across notable subsets of the data, which is not exposed by a low average error for the entire dataset. Distributionally Robust Optimization (DRO) seemingly addresses this problem by minimizing the worst expected risk across subpopulations. We establish theoretical results that clarify the relation between DRO and the optimization of the same loss averaged on an adequately weighted training dataset. The results cover finite and infinite number of training distributions, as well as convex and non-convex loss functions. An implication of our results is that for each DRO problem there exists a data distribution such that learning this distribution is equivalent to solving the DRO problem. Yet, important problems that DRO seeks to address (for instance, adversarial robustness and fighting bias) cannot be reduced to finding the one ’unbiased’ dataset. Our discussion section addresses this important discrepancy.
APA
Słowik, A. & Bottou, L.. (2022). On Distributionally Robust Optimization and Data Rebalancing . Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 151:1283-1297 Available from https://proceedings.mlr.press/v151/slowik22a.html.

Related Material