A Bayesian Wilcoxon signed-rank test based on the Dirichlet process

Alessio Benavoli, Giorgio Corani, Francesca Mangili, Marco Zaffalon, Fabrizio Ruggeri
Proceedings of the 31st International Conference on Machine Learning, PMLR 32(2):1026-1034, 2014.

Abstract

Bayesian methods are ubiquitous in machine learning. Nevertheless, the analysis of empirical results is typically performed by frequentist tests. This implies dealing with null hypothesis significance tests and p-values, even though the shortcomings of such methods are well known. We propose a nonparametric Bayesian version of the Wilcoxon signed-rank test using a Dirichlet process (DP) based prior. We address in two different ways the problem of how to choose the infinite dimensional parameter that characterizes the DP. The proposed test has all the traditional strengths of the Bayesian approach; for instance, unlike the frequentist tests, it allows verifying the null hypothesis, not only rejecting it, and taking decision which minimize the expected loss. Moreover, one of the solutions proposed to model the infinitedimensional parameter of the DP, allows isolating instances in which the traditional frequentist test is guessing at random. We show results dealing with the comparison of two classifiers using real and simulated data.

Cite this Paper

BibTeX
@InProceedings{pmlr-v32-benavoli14, title = {A Bayesian Wilcoxon signed-rank test based on the Dirichlet process}, author = {Benavoli, Alessio and Corani, Giorgio and Mangili, Francesca and Zaffalon, Marco and Ruggeri, Fabrizio}, booktitle = {Proceedings of the 31st International Conference on Machine Learning}, pages = {1026--1034}, year = {2014}, editor = {Xing, Eric P. and Jebara, Tony}, volume = {32}, number = {2}, series = {Proceedings of Machine Learning Research}, address = {Bejing, China}, month = {22--24 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v32/benavoli14.pdf}, url = {https://proceedings.mlr.press/v32/benavoli14.html}, abstract = {Bayesian methods are ubiquitous in machine learning. Nevertheless, the analysis of empirical results is typically performed by frequentist tests. This implies dealing with null hypothesis significance tests and p-values, even though the shortcomings of such methods are well known. We propose a nonparametric Bayesian version of the Wilcoxon signed-rank test using a Dirichlet process (DP) based prior. We address in two different ways the problem of how to choose the infinite dimensional parameter that characterizes the DP. The proposed test has all the traditional strengths of the Bayesian approach; for instance, unlike the frequentist tests, it allows verifying the null hypothesis, not only rejecting it, and taking decision which minimize the expected loss. Moreover, one of the solutions proposed to model the infinitedimensional parameter of the DP, allows isolating instances in which the traditional frequentist test is guessing at random. We show results dealing with the comparison of two classifiers using real and simulated data.} }
Endnote
%0 Conference Paper %T A Bayesian Wilcoxon signed-rank test based on the Dirichlet process %A Alessio Benavoli %A Giorgio Corani %A Francesca Mangili %A Marco Zaffalon %A Fabrizio Ruggeri %B Proceedings of the 31st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2014 %E Eric P. Xing %E Tony Jebara %F pmlr-v32-benavoli14 %I PMLR %P 1026--1034 %U https://proceedings.mlr.press/v32/benavoli14.html %V 32 %N 2 %X Bayesian methods are ubiquitous in machine learning. Nevertheless, the analysis of empirical results is typically performed by frequentist tests. This implies dealing with null hypothesis significance tests and p-values, even though the shortcomings of such methods are well known. We propose a nonparametric Bayesian version of the Wilcoxon signed-rank test using a Dirichlet process (DP) based prior. We address in two different ways the problem of how to choose the infinite dimensional parameter that characterizes the DP. The proposed test has all the traditional strengths of the Bayesian approach; for instance, unlike the frequentist tests, it allows verifying the null hypothesis, not only rejecting it, and taking decision which minimize the expected loss. Moreover, one of the solutions proposed to model the infinitedimensional parameter of the DP, allows isolating instances in which the traditional frequentist test is guessing at random. We show results dealing with the comparison of two classifiers using real and simulated data.
RIS
TY - CPAPER TI - A Bayesian Wilcoxon signed-rank test based on the Dirichlet process AU - Alessio Benavoli AU - Giorgio Corani AU - Francesca Mangili AU - Marco Zaffalon AU - Fabrizio Ruggeri BT - Proceedings of the 31st International Conference on Machine Learning DA - 2014/06/18 ED - Eric P. Xing ED - Tony Jebara ID - pmlr-v32-benavoli14 PB - PMLR DP - Proceedings of Machine Learning Research VL - 32 IS - 2 SP - 1026 EP - 1034 L1 - http://proceedings.mlr.press/v32/benavoli14.pdf UR - https://proceedings.mlr.press/v32/benavoli14.html AB - Bayesian methods are ubiquitous in machine learning. Nevertheless, the analysis of empirical results is typically performed by frequentist tests. This implies dealing with null hypothesis significance tests and p-values, even though the shortcomings of such methods are well known. We propose a nonparametric Bayesian version of the Wilcoxon signed-rank test using a Dirichlet process (DP) based prior. We address in two different ways the problem of how to choose the infinite dimensional parameter that characterizes the DP. The proposed test has all the traditional strengths of the Bayesian approach; for instance, unlike the frequentist tests, it allows verifying the null hypothesis, not only rejecting it, and taking decision which minimize the expected loss. Moreover, one of the solutions proposed to model the infinitedimensional parameter of the DP, allows isolating instances in which the traditional frequentist test is guessing at random. We show results dealing with the comparison of two classifiers using real and simulated data. ER -
APA
Benavoli, A., Corani, G., Mangili, F., Zaffalon, M. & Ruggeri, F.. (2014). A Bayesian Wilcoxon signed-rank test based on the Dirichlet process. Proceedings of the 31st International Conference on Machine Learning, in Proceedings of Machine Learning Research 32(2):1026-1034 Available from https://proceedings.mlr.press/v32/benavoli14.html.

Related Material