Comparing Prequential Model Selection Criteria in Supervised Learning of Mixture Models

Petri Kontkanen, Petri Myllymäki, Henry Tirri
Proceedings of the Eighth International Workshop on Artificial Intelligence and Statistics, PMLR R3:156-161, 2001.

Abstract

In this paper we study prequential model selection criteria in supervised learning domains. The main problem with this approach is the fact that the criterion is sensitive to the ordering the data is processed with. We discuss several approaches for addressing the ordering problem, and compare empirically their performance in real-world supervised model selection tasks. The empirical results demonstrate that with the prequential approach it is quite easy to find predictive models that are significantly more accurate classifiers than the models found by the standard unsupervised marginal likelihood criterion. The results also suggest that averaging over random orderings may be a more sensible strategy for solving the ordering problem than trying to find the ordering optimizing the prequential model selection criterion.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR3-kontkanen01a, title = {Comparing Prequential Model Selection Criteria in Supervised Learning of Mixture Models}, author = {Kontkanen, Petri and Myllym{\"{a}}ki, Petri and Tirri, Henry}, booktitle = {Proceedings of the Eighth International Workshop on Artificial Intelligence and Statistics}, pages = {156--161}, year = {2001}, editor = {Richardson, Thomas S. and Jaakkola, Tommi S.}, volume = {R3}, series = {Proceedings of Machine Learning Research}, month = {04--07 Jan}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/r3/kontkanen01a/kontkanen01a.pdf}, url = {https://proceedings.mlr.press/r3/kontkanen01a.html}, abstract = {In this paper we study prequential model selection criteria in supervised learning domains. The main problem with this approach is the fact that the criterion is sensitive to the ordering the data is processed with. We discuss several approaches for addressing the ordering problem, and compare empirically their performance in real-world supervised model selection tasks. The empirical results demonstrate that with the prequential approach it is quite easy to find predictive models that are significantly more accurate classifiers than the models found by the standard unsupervised marginal likelihood criterion. The results also suggest that averaging over random orderings may be a more sensible strategy for solving the ordering problem than trying to find the ordering optimizing the prequential model selection criterion.}, note = {Reissued by PMLR on 31 March 2021.} }
Endnote
%0 Conference Paper %T Comparing Prequential Model Selection Criteria in Supervised Learning of Mixture Models %A Petri Kontkanen %A Petri Myllymäki %A Henry Tirri %B Proceedings of the Eighth International Workshop on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2001 %E Thomas S. Richardson %E Tommi S. Jaakkola %F pmlr-vR3-kontkanen01a %I PMLR %P 156--161 %U https://proceedings.mlr.press/r3/kontkanen01a.html %V R3 %X In this paper we study prequential model selection criteria in supervised learning domains. The main problem with this approach is the fact that the criterion is sensitive to the ordering the data is processed with. We discuss several approaches for addressing the ordering problem, and compare empirically their performance in real-world supervised model selection tasks. The empirical results demonstrate that with the prequential approach it is quite easy to find predictive models that are significantly more accurate classifiers than the models found by the standard unsupervised marginal likelihood criterion. The results also suggest that averaging over random orderings may be a more sensible strategy for solving the ordering problem than trying to find the ordering optimizing the prequential model selection criterion. %Z Reissued by PMLR on 31 March 2021.
APA
Kontkanen, P., Myllymäki, P. & Tirri, H.. (2001). Comparing Prequential Model Selection Criteria in Supervised Learning of Mixture Models. Proceedings of the Eighth International Workshop on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research R3:156-161 Available from https://proceedings.mlr.press/r3/kontkanen01a.html. Reissued by PMLR on 31 March 2021.

Related Material