PAC-Bayesian Generalization Bound for Density Estimation with Application to Co-clustering

Yevgeny Seldin, Naftali Tishby
Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics, PMLR 5:472-479, 2009.

Abstract

We derive a PAC-Bayesian generalization bound for density estimation. Similar to the PAC-Bayesian generalization bound for classification, the result has the appealingly simple form of a tradeoff between empirical performance and the KL-divergence of the posterior from the prior. Moreover, the PAC-Bayesian generalization bound for classification can be derived as a special case of the bound for density estimation. To illustrate a possible application of our bound we derive a generalization bound for co-clustering. The bound provides a criterion to evaluate the ability of co-clustering to predict new co-occurrences, thus introducing a supervised flavor to this traditionally unsupervised task.

Cite this Paper

BibTeX
@InProceedings{pmlr-v5-seldin09a, title = {PAC-Bayesian Generalization Bound for Density Estimation with Application to Co-clustering}, author = {Seldin, Yevgeny and Tishby, Naftali}, booktitle = {Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics}, pages = {472--479}, year = {2009}, editor = {van Dyk, David and Welling, Max}, volume = {5}, series = {Proceedings of Machine Learning Research}, address = {Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA}, month = {16--18 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v5/seldin09a/seldin09a.pdf}, url = {https://proceedings.mlr.press/v5/seldin09a.html}, abstract = {We derive a PAC-Bayesian generalization bound for density estimation. Similar to the PAC-Bayesian generalization bound for classification, the result has the appealingly simple form of a tradeoff between empirical performance and the KL-divergence of the posterior from the prior. Moreover, the PAC-Bayesian generalization bound for classification can be derived as a special case of the bound for density estimation. To illustrate a possible application of our bound we derive a generalization bound for co-clustering. The bound provides a criterion to evaluate the ability of co-clustering to predict new co-occurrences, thus introducing a supervised flavor to this traditionally unsupervised task.} }
Endnote
%0 Conference Paper %T PAC-Bayesian Generalization Bound for Density Estimation with Application to Co-clustering %A Yevgeny Seldin %A Naftali Tishby %B Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2009 %E David van Dyk %E Max Welling %F pmlr-v5-seldin09a %I PMLR %P 472--479 %U https://proceedings.mlr.press/v5/seldin09a.html %V 5 %X We derive a PAC-Bayesian generalization bound for density estimation. Similar to the PAC-Bayesian generalization bound for classification, the result has the appealingly simple form of a tradeoff between empirical performance and the KL-divergence of the posterior from the prior. Moreover, the PAC-Bayesian generalization bound for classification can be derived as a special case of the bound for density estimation. To illustrate a possible application of our bound we derive a generalization bound for co-clustering. The bound provides a criterion to evaluate the ability of co-clustering to predict new co-occurrences, thus introducing a supervised flavor to this traditionally unsupervised task.
RIS
TY - CPAPER TI - PAC-Bayesian Generalization Bound for Density Estimation with Application to Co-clustering AU - Yevgeny Seldin AU - Naftali Tishby BT - Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics DA - 2009/04/15 ED - David van Dyk ED - Max Welling ID - pmlr-v5-seldin09a PB - PMLR DP - Proceedings of Machine Learning Research VL - 5 SP - 472 EP - 479 L1 - http://proceedings.mlr.press/v5/seldin09a/seldin09a.pdf UR - https://proceedings.mlr.press/v5/seldin09a.html AB - We derive a PAC-Bayesian generalization bound for density estimation. Similar to the PAC-Bayesian generalization bound for classification, the result has the appealingly simple form of a tradeoff between empirical performance and the KL-divergence of the posterior from the prior. Moreover, the PAC-Bayesian generalization bound for classification can be derived as a special case of the bound for density estimation. To illustrate a possible application of our bound we derive a generalization bound for co-clustering. The bound provides a criterion to evaluate the ability of co-clustering to predict new co-occurrences, thus introducing a supervised flavor to this traditionally unsupervised task. ER -
APA
Seldin, Y. & Tishby, N.. (2009). PAC-Bayesian Generalization Bound for Density Estimation with Application to Co-clustering. Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 5:472-479 Available from https://proceedings.mlr.press/v5/seldin09a.html.

Related Material