Estimation Beyond Data Reweighting: Kernel Method of Moments

Heiner Kremer, Yassine Nemmour, Bernhard Schölkopf, Jia-Jie Zhu
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:17745-17783, 2023.

Abstract

Moment restrictions and their conditional counterparts emerge in many areas of machine learning and statistics ranging from causal inference to reinforcement learning. Estimators for these tasks, generally called methods of moments, include the prominent generalized method of moments (GMM) which has recently gained attention in causal inference. GMM is a special case of the broader family of empirical likelihood estimators which are based on approximating a population distribution by means of minimizing a $\varphi$-divergence to an empirical distribution. However, the use of $\varphi$-divergences effectively limits the candidate distributions to reweightings of the data samples. We lift this long-standing limitation and provide a method of moments that goes beyond data reweighting. This is achieved by defining an empirical likelihood estimator based on maximum mean discrepancy which we term the kernel method of moments (KMM). We provide a variant of our estimator for conditional moment restrictions and show that it is asymptotically first-order optimal for such problems. Finally, we show that our method achieves competitive performance on several conditional moment restriction tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-kremer23a, title = {Estimation Beyond Data Reweighting: Kernel Method of Moments}, author = {Kremer, Heiner and Nemmour, Yassine and Sch\"{o}lkopf, Bernhard and Zhu, Jia-Jie}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {17745--17783}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/kremer23a/kremer23a.pdf}, url = {https://proceedings.mlr.press/v202/kremer23a.html}, abstract = {Moment restrictions and their conditional counterparts emerge in many areas of machine learning and statistics ranging from causal inference to reinforcement learning. Estimators for these tasks, generally called methods of moments, include the prominent generalized method of moments (GMM) which has recently gained attention in causal inference. GMM is a special case of the broader family of empirical likelihood estimators which are based on approximating a population distribution by means of minimizing a $\varphi$-divergence to an empirical distribution. However, the use of $\varphi$-divergences effectively limits the candidate distributions to reweightings of the data samples. We lift this long-standing limitation and provide a method of moments that goes beyond data reweighting. This is achieved by defining an empirical likelihood estimator based on maximum mean discrepancy which we term the kernel method of moments (KMM). We provide a variant of our estimator for conditional moment restrictions and show that it is asymptotically first-order optimal for such problems. Finally, we show that our method achieves competitive performance on several conditional moment restriction tasks.} }
Endnote
%0 Conference Paper %T Estimation Beyond Data Reweighting: Kernel Method of Moments %A Heiner Kremer %A Yassine Nemmour %A Bernhard Schölkopf %A Jia-Jie Zhu %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-kremer23a %I PMLR %P 17745--17783 %U https://proceedings.mlr.press/v202/kremer23a.html %V 202 %X Moment restrictions and their conditional counterparts emerge in many areas of machine learning and statistics ranging from causal inference to reinforcement learning. Estimators for these tasks, generally called methods of moments, include the prominent generalized method of moments (GMM) which has recently gained attention in causal inference. GMM is a special case of the broader family of empirical likelihood estimators which are based on approximating a population distribution by means of minimizing a $\varphi$-divergence to an empirical distribution. However, the use of $\varphi$-divergences effectively limits the candidate distributions to reweightings of the data samples. We lift this long-standing limitation and provide a method of moments that goes beyond data reweighting. This is achieved by defining an empirical likelihood estimator based on maximum mean discrepancy which we term the kernel method of moments (KMM). We provide a variant of our estimator for conditional moment restrictions and show that it is asymptotically first-order optimal for such problems. Finally, we show that our method achieves competitive performance on several conditional moment restriction tasks.
APA
Kremer, H., Nemmour, Y., Schölkopf, B. & Zhu, J.. (2023). Estimation Beyond Data Reweighting: Kernel Method of Moments. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:17745-17783 Available from https://proceedings.mlr.press/v202/kremer23a.html.

Related Material