Validation of Approximate Likelihood and Emulator Models for Computationally Intensive Simulations

Niccolo Dalmasso, Ann Lee, Rafael Izbicki, Taylor Pospisil, Ilmun Kim, Chieh-An Lin
Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, PMLR 108:3349-3361, 2020.

Abstract

Complex phenomena in engineering and the sciences are often modeled with computationally intensive feed-forward simulations for which a tractable analytic likelihood does not exist. In these cases, it is sometimes necessary to estimate an approximate likelihood or fit a fast emulator model for efficient statistical inference; such surrogate models include Gaussian synthetic likelihoods and more recently neural density estimators such as autoregressive models and normalizing flows. To date, however, there is no consistent way of quantifying the quality of such a fit. Here we propose a statistical framework that can distinguish any arbitrary misspecified model from the target likelihood, and that in addition can identify with statistical confidence the regions of parameter as well as feature space where the fit is inadequate. At the heart of our approach is a two-sample test that quantifies the quality of the fit at fixed parameter values, and a global test that assesses goodness-of-fit across simulation parameters. While our general framework can incorporate any test statistic or distance metric, we specifically argue for a new two-sample test that can leverage any regression method to attain high power and provide diagnostics in complex data settings. Software for our approach is available on GitHub in Python and R.

Cite this Paper

BibTeX
@InProceedings{pmlr-v108-dalmasso20a, title = {Validation of Approximate Likelihood and Emulator Models for Computationally Intensive Simulations}, author = {Dalmasso, Niccolo and Lee, Ann and Izbicki, Rafael and Pospisil, Taylor and Kim, Ilmun and Lin, Chieh-An}, booktitle = {Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics}, pages = {3349--3361}, year = {2020}, editor = {Chiappa, Silvia and Calandra, Roberto}, volume = {108}, series = {Proceedings of Machine Learning Research}, month = {26--28 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v108/dalmasso20a/dalmasso20a.pdf}, url = {https://proceedings.mlr.press/v108/dalmasso20a.html}, abstract = {Complex phenomena in engineering and the sciences are often modeled with computationally intensive feed-forward simulations for which a tractable analytic likelihood does not exist. In these cases, it is sometimes necessary to estimate an approximate likelihood or fit a fast emulator model for efficient statistical inference; such surrogate models include Gaussian synthetic likelihoods and more recently neural density estimators such as autoregressive models and normalizing flows. To date, however, there is no consistent way of quantifying the quality of such a fit. Here we propose a statistical framework that can distinguish any arbitrary misspecified model from the target likelihood, and that in addition can identify with statistical confidence the regions of parameter as well as feature space where the fit is inadequate. At the heart of our approach is a two-sample test that quantifies the quality of the fit at fixed parameter values, and a global test that assesses goodness-of-fit across simulation parameters. While our general framework can incorporate any test statistic or distance metric, we specifically argue for a new two-sample test that can leverage any regression method to attain high power and provide diagnostics in complex data settings. Software for our approach is available on GitHub in Python and R. } }
Endnote
%0 Conference Paper %T Validation of Approximate Likelihood and Emulator Models for Computationally Intensive Simulations %A Niccolo Dalmasso %A Ann Lee %A Rafael Izbicki %A Taylor Pospisil %A Ilmun Kim %A Chieh-An Lin %B Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2020 %E Silvia Chiappa %E Roberto Calandra %F pmlr-v108-dalmasso20a %I PMLR %P 3349--3361 %U https://proceedings.mlr.press/v108/dalmasso20a.html %V 108 %X Complex phenomena in engineering and the sciences are often modeled with computationally intensive feed-forward simulations for which a tractable analytic likelihood does not exist. In these cases, it is sometimes necessary to estimate an approximate likelihood or fit a fast emulator model for efficient statistical inference; such surrogate models include Gaussian synthetic likelihoods and more recently neural density estimators such as autoregressive models and normalizing flows. To date, however, there is no consistent way of quantifying the quality of such a fit. Here we propose a statistical framework that can distinguish any arbitrary misspecified model from the target likelihood, and that in addition can identify with statistical confidence the regions of parameter as well as feature space where the fit is inadequate. At the heart of our approach is a two-sample test that quantifies the quality of the fit at fixed parameter values, and a global test that assesses goodness-of-fit across simulation parameters. While our general framework can incorporate any test statistic or distance metric, we specifically argue for a new two-sample test that can leverage any regression method to attain high power and provide diagnostics in complex data settings. Software for our approach is available on GitHub in Python and R.
APA
Dalmasso, N., Lee, A., Izbicki, R., Pospisil, T., Kim, I. & Lin, C.. (2020). Validation of Approximate Likelihood and Emulator Models for Computationally Intensive Simulations. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 108:3349-3361 Available from https://proceedings.mlr.press/v108/dalmasso20a.html.

Related Material