Hyperparameter Tuning for Causal Inference with Double Machine Learning: A Simulation Study

Philipp Bach, Oliver Schacht, Victor Chernozhukov, Sven Klaassen, Martin Spindler
Proceedings of the Third Conference on Causal Learning and Reasoning, PMLR 236:1065-1117, 2024.

Abstract

Proper hyperparameter tuning is essential for achieving optimal performance of modern machine learning (ML) methods in predictive tasks. While there is an extensive literature on tuning ML learners for prediction, there is only littleguidance available on tuning ML learners for causal machine learning. In this paper, we investigate the role of hyperparameter tuning and other practical decisions for causal estimation as based on the Double Machine Learning approach by Chernozhukov et al. (2018). Double Machine Learning relies on estimating so-called nuisance parameters by treating them as supervised learning problems and using them as plug-in estimates to solve for the (causal) parameter. We conduct an extensive simulation study using data from the 2019 Atlantic Causal Inference Conference Data Challenge. We provide empirical insights on the selection and calibration of the ML methods for the performance of causal estimation. First, we assess the importance of data splitting schemes for tuning ML learners within Double Machine Learning. Second, we investigate the choice of ML methods and hyperparameters, including recent AutoML frameworks, and consider the relationship between their predictive power and the estimation performance for a causal parameter of interest. Third, we investigate to what extent the choice of a particular causal model, as characterized by incorporated parametric assumptions, can be based on predictive performance metrics.

Cite this Paper

BibTeX
@InProceedings{pmlr-v236-bach24a, title = {Hyperparameter Tuning for Causal Inference with Double Machine Learning: A Simulation Study}, author = {Bach, Philipp and Schacht, Oliver and Chernozhukov, Victor and Klaassen, Sven and Spindler, Martin}, booktitle = {Proceedings of the Third Conference on Causal Learning and Reasoning}, pages = {1065--1117}, year = {2024}, editor = {Locatello, Francesco and Didelez, Vanessa}, volume = {236}, series = {Proceedings of Machine Learning Research}, month = {01--03 Apr}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v236/bach24a/bach24a.pdf}, url = {https://proceedings.mlr.press/v236/bach24a.html}, abstract = {Proper hyperparameter tuning is essential for achieving optimal performance of modern machine learning (ML) methods in predictive tasks. While there is an extensive literature on tuning ML learners for prediction, there is only littleguidance available on tuning ML learners for causal machine learning. In this paper, we investigate the role of hyperparameter tuning and other practical decisions for causal estimation as based on the Double Machine Learning approach by Chernozhukov et al. (2018). Double Machine Learning relies on estimating so-called nuisance parameters by treating them as supervised learning problems and using them as plug-in estimates to solve for the (causal) parameter. We conduct an extensive simulation study using data from the 2019 Atlantic Causal Inference Conference Data Challenge. We provide empirical insights on the selection and calibration of the ML methods for the performance of causal estimation. First, we assess the importance of data splitting schemes for tuning ML learners within Double Machine Learning. Second, we investigate the choice of ML methods and hyperparameters, including recent AutoML frameworks, and consider the relationship between their predictive power and the estimation performance for a causal parameter of interest. Third, we investigate to what extent the choice of a particular causal model, as characterized by incorporated parametric assumptions, can be based on predictive performance metrics.} }
Endnote
%0 Conference Paper %T Hyperparameter Tuning for Causal Inference with Double Machine Learning: A Simulation Study %A Philipp Bach %A Oliver Schacht %A Victor Chernozhukov %A Sven Klaassen %A Martin Spindler %B Proceedings of the Third Conference on Causal Learning and Reasoning %C Proceedings of Machine Learning Research %D 2024 %E Francesco Locatello %E Vanessa Didelez %F pmlr-v236-bach24a %I PMLR %P 1065--1117 %U https://proceedings.mlr.press/v236/bach24a.html %V 236 %X Proper hyperparameter tuning is essential for achieving optimal performance of modern machine learning (ML) methods in predictive tasks. While there is an extensive literature on tuning ML learners for prediction, there is only littleguidance available on tuning ML learners for causal machine learning. In this paper, we investigate the role of hyperparameter tuning and other practical decisions for causal estimation as based on the Double Machine Learning approach by Chernozhukov et al. (2018). Double Machine Learning relies on estimating so-called nuisance parameters by treating them as supervised learning problems and using them as plug-in estimates to solve for the (causal) parameter. We conduct an extensive simulation study using data from the 2019 Atlantic Causal Inference Conference Data Challenge. We provide empirical insights on the selection and calibration of the ML methods for the performance of causal estimation. First, we assess the importance of data splitting schemes for tuning ML learners within Double Machine Learning. Second, we investigate the choice of ML methods and hyperparameters, including recent AutoML frameworks, and consider the relationship between their predictive power and the estimation performance for a causal parameter of interest. Third, we investigate to what extent the choice of a particular causal model, as characterized by incorporated parametric assumptions, can be based on predictive performance metrics.
APA
Bach, P., Schacht, O., Chernozhukov, V., Klaassen, S. & Spindler, M.. (2024). Hyperparameter Tuning for Causal Inference with Double Machine Learning: A Simulation Study. Proceedings of the Third Conference on Causal Learning and Reasoning, in Proceedings of Machine Learning Research 236:1065-1117 Available from https://proceedings.mlr.press/v236/bach24a.html.

Related Material