Evaluating Bayesian Models with Posterior Dispersion Indices

Alp Kucukelbir, Yixin Wang, David M. Blei
Proceedings of the 34th International Conference on Machine Learning, PMLR 70:1925-1934, 2017.

Abstract

Probabilistic modeling is cyclical: we specify a model, infer its posterior, and evaluate its performance. Evaluation drives the cycle, as we revise our model based on how it performs. This requires a metric. Traditionally, predictive accuracy prevails. Yet, predictive accuracy does not tell the whole story. We propose to evaluate a model through posterior dispersion. The idea is to analyze how each datapoint fares in relation to posterior uncertainty around the hidden structure. This highlights datapoints the model struggles to explain and provides complimentary insight to datapoints with low predictive accuracy. We present a family of posterior dispersion indices (PDI) that capture this idea. We show how a PDI identifies patterns of model mismatch in three real data examples: voting preferences, supermarket shopping, and population genetics.

Cite this Paper

BibTeX
@InProceedings{pmlr-v70-kucukelbir17a, title = {Evaluating {B}ayesian Models with Posterior Dispersion Indices}, author = {Alp Kucukelbir and Yixin Wang and David M. Blei}, booktitle = {Proceedings of the 34th International Conference on Machine Learning}, pages = {1925--1934}, year = {2017}, editor = {Precup, Doina and Teh, Yee Whye}, volume = {70}, series = {Proceedings of Machine Learning Research}, month = {06--11 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v70/kucukelbir17a/kucukelbir17a.pdf}, url = {https://proceedings.mlr.press/v70/kucukelbir17a.html}, abstract = {Probabilistic modeling is cyclical: we specify a model, infer its posterior, and evaluate its performance. Evaluation drives the cycle, as we revise our model based on how it performs. This requires a metric. Traditionally, predictive accuracy prevails. Yet, predictive accuracy does not tell the whole story. We propose to evaluate a model through posterior dispersion. The idea is to analyze how each datapoint fares in relation to posterior uncertainty around the hidden structure. This highlights datapoints the model struggles to explain and provides complimentary insight to datapoints with low predictive accuracy. We present a family of posterior dispersion indices (PDI) that capture this idea. We show how a PDI identifies patterns of model mismatch in three real data examples: voting preferences, supermarket shopping, and population genetics.} }
Endnote
%0 Conference Paper %T Evaluating Bayesian Models with Posterior Dispersion Indices %A Alp Kucukelbir %A Yixin Wang %A David M. Blei %B Proceedings of the 34th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2017 %E Doina Precup %E Yee Whye Teh %F pmlr-v70-kucukelbir17a %I PMLR %P 1925--1934 %U https://proceedings.mlr.press/v70/kucukelbir17a.html %V 70 %X Probabilistic modeling is cyclical: we specify a model, infer its posterior, and evaluate its performance. Evaluation drives the cycle, as we revise our model based on how it performs. This requires a metric. Traditionally, predictive accuracy prevails. Yet, predictive accuracy does not tell the whole story. We propose to evaluate a model through posterior dispersion. The idea is to analyze how each datapoint fares in relation to posterior uncertainty around the hidden structure. This highlights datapoints the model struggles to explain and provides complimentary insight to datapoints with low predictive accuracy. We present a family of posterior dispersion indices (PDI) that capture this idea. We show how a PDI identifies patterns of model mismatch in three real data examples: voting preferences, supermarket shopping, and population genetics.
APA
Kucukelbir, A., Wang, Y. & Blei, D.M.. (2017). Evaluating Bayesian Models with Posterior Dispersion Indices. Proceedings of the 34th International Conference on Machine Learning, in Proceedings of Machine Learning Research 70:1925-1934 Available from https://proceedings.mlr.press/v70/kucukelbir17a.html.

Related Material