Heterogeneous Risk Minimization

Jiashuo Liu, Zheyuan Hu, Peng Cui, Bo Li, Zheyan Shen
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:6804-6814, 2021.

Abstract

Machine learning algorithms with empirical risk minimization usually suffer from poor generalization performance due to the greedy exploitation of correlations among the training data, which are not stable under distributional shifts. Recently, some invariant learning methods for out-of-distribution (OOD) generalization have been proposed by leveraging multiple training environments to find invariant relationships. However, modern datasets are frequently assembled by merging data from multiple sources without explicit source labels. The resultant unobserved heterogeneity renders many invariant learning methods inapplicable. In this paper, we propose Heterogeneous Risk Minimization (HRM) framework to achieve joint learning of latent heterogeneity among the data and invariant relationship, which leads to stable prediction despite distributional shifts. We theoretically characterize the roles of the environment labels in invariant learning and justify our newly proposed HRM framework. Extensive experimental results validate the effectiveness of our HRM framework.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-liu21h, title = {Heterogeneous Risk Minimization}, author = {Liu, Jiashuo and Hu, Zheyuan and Cui, Peng and Li, Bo and Shen, Zheyan}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {6804--6814}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/liu21h/liu21h.pdf}, url = {https://proceedings.mlr.press/v139/liu21h.html}, abstract = {Machine learning algorithms with empirical risk minimization usually suffer from poor generalization performance due to the greedy exploitation of correlations among the training data, which are not stable under distributional shifts. Recently, some invariant learning methods for out-of-distribution (OOD) generalization have been proposed by leveraging multiple training environments to find invariant relationships. However, modern datasets are frequently assembled by merging data from multiple sources without explicit source labels. The resultant unobserved heterogeneity renders many invariant learning methods inapplicable. In this paper, we propose Heterogeneous Risk Minimization (HRM) framework to achieve joint learning of latent heterogeneity among the data and invariant relationship, which leads to stable prediction despite distributional shifts. We theoretically characterize the roles of the environment labels in invariant learning and justify our newly proposed HRM framework. Extensive experimental results validate the effectiveness of our HRM framework.} }
Endnote
%0 Conference Paper %T Heterogeneous Risk Minimization %A Jiashuo Liu %A Zheyuan Hu %A Peng Cui %A Bo Li %A Zheyan Shen %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-liu21h %I PMLR %P 6804--6814 %U https://proceedings.mlr.press/v139/liu21h.html %V 139 %X Machine learning algorithms with empirical risk minimization usually suffer from poor generalization performance due to the greedy exploitation of correlations among the training data, which are not stable under distributional shifts. Recently, some invariant learning methods for out-of-distribution (OOD) generalization have been proposed by leveraging multiple training environments to find invariant relationships. However, modern datasets are frequently assembled by merging data from multiple sources without explicit source labels. The resultant unobserved heterogeneity renders many invariant learning methods inapplicable. In this paper, we propose Heterogeneous Risk Minimization (HRM) framework to achieve joint learning of latent heterogeneity among the data and invariant relationship, which leads to stable prediction despite distributional shifts. We theoretically characterize the roles of the environment labels in invariant learning and justify our newly proposed HRM framework. Extensive experimental results validate the effectiveness of our HRM framework.
APA
Liu, J., Hu, Z., Cui, P., Li, B. & Shen, Z.. (2021). Heterogeneous Risk Minimization. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:6804-6814 Available from https://proceedings.mlr.press/v139/liu21h.html.

Related Material