Analysis of Empirical MAP and Empirical Partially Bayes: Can They be Alternatives to Variational Bayes?

Shinichi Nakajima, Masashi Sugiyama
Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, PMLR 33:20-28, 2014.

Abstract

Variational Bayesian (VB) learning is known to be a promising approximation to Bayesian learning with computational efficiency. However, in some applications, e.g., large-scale collaborative filtering and tensor factorization, VB is still computationally too costly. In such cases, looser approximations such as MAP estimation and partially Bayesian (PB) learning, where a part of the parameters are point-estimated, seem attractive. In this paper, we theoretically investigate the behavior of the MAP and the PB solutions of matrix factorization. A notable finding is that the global solutions of MAP and PB in the empirical Bayesian scenario, where the hyperparameters are also estimated from observation, are trivial and useless, while their local solutions behave similarly to the global solution of VB. This suggests that empirical MAP and empirical PB with local search can be alternatives to empirical VB equipped with the useful automatic relevance determination property. Experiments support our theory.

Cite this Paper

BibTeX
@InProceedings{pmlr-v33-nakajima14, title = {{Analysis of Empirical MAP and Empirical Partially Bayes: Can They be Alternatives to Variational Bayes?}}, author = {Nakajima, Shinichi and Sugiyama, Masashi}, booktitle = {Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics}, pages = {20--28}, year = {2014}, editor = {Kaski, Samuel and Corander, Jukka}, volume = {33}, series = {Proceedings of Machine Learning Research}, address = {Reykjavik, Iceland}, month = {22--25 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v33/nakajima14.pdf}, url = {https://proceedings.mlr.press/v33/nakajima14.html}, abstract = {Variational Bayesian (VB) learning is known to be a promising approximation to Bayesian learning with computational efficiency. However, in some applications, e.g., large-scale collaborative filtering and tensor factorization, VB is still computationally too costly. In such cases, looser approximations such as MAP estimation and partially Bayesian (PB) learning, where a part of the parameters are point-estimated, seem attractive. In this paper, we theoretically investigate the behavior of the MAP and the PB solutions of matrix factorization. A notable finding is that the global solutions of MAP and PB in the empirical Bayesian scenario, where the hyperparameters are also estimated from observation, are trivial and useless, while their local solutions behave similarly to the global solution of VB. This suggests that empirical MAP and empirical PB with local search can be alternatives to empirical VB equipped with the useful automatic relevance determination property. Experiments support our theory.} }
Endnote
%0 Conference Paper %T Analysis of Empirical MAP and Empirical Partially Bayes: Can They be Alternatives to Variational Bayes? %A Shinichi Nakajima %A Masashi Sugiyama %B Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2014 %E Samuel Kaski %E Jukka Corander %F pmlr-v33-nakajima14 %I PMLR %P 20--28 %U https://proceedings.mlr.press/v33/nakajima14.html %V 33 %X Variational Bayesian (VB) learning is known to be a promising approximation to Bayesian learning with computational efficiency. However, in some applications, e.g., large-scale collaborative filtering and tensor factorization, VB is still computationally too costly. In such cases, looser approximations such as MAP estimation and partially Bayesian (PB) learning, where a part of the parameters are point-estimated, seem attractive. In this paper, we theoretically investigate the behavior of the MAP and the PB solutions of matrix factorization. A notable finding is that the global solutions of MAP and PB in the empirical Bayesian scenario, where the hyperparameters are also estimated from observation, are trivial and useless, while their local solutions behave similarly to the global solution of VB. This suggests that empirical MAP and empirical PB with local search can be alternatives to empirical VB equipped with the useful automatic relevance determination property. Experiments support our theory.
RIS
TY - CPAPER TI - Analysis of Empirical MAP and Empirical Partially Bayes: Can They be Alternatives to Variational Bayes? AU - Shinichi Nakajima AU - Masashi Sugiyama BT - Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics DA - 2014/04/02 ED - Samuel Kaski ED - Jukka Corander ID - pmlr-v33-nakajima14 PB - PMLR DP - Proceedings of Machine Learning Research VL - 33 SP - 20 EP - 28 L1 - http://proceedings.mlr.press/v33/nakajima14.pdf UR - https://proceedings.mlr.press/v33/nakajima14.html AB - Variational Bayesian (VB) learning is known to be a promising approximation to Bayesian learning with computational efficiency. However, in some applications, e.g., large-scale collaborative filtering and tensor factorization, VB is still computationally too costly. In such cases, looser approximations such as MAP estimation and partially Bayesian (PB) learning, where a part of the parameters are point-estimated, seem attractive. In this paper, we theoretically investigate the behavior of the MAP and the PB solutions of matrix factorization. A notable finding is that the global solutions of MAP and PB in the empirical Bayesian scenario, where the hyperparameters are also estimated from observation, are trivial and useless, while their local solutions behave similarly to the global solution of VB. This suggests that empirical MAP and empirical PB with local search can be alternatives to empirical VB equipped with the useful automatic relevance determination property. Experiments support our theory. ER -
APA
Nakajima, S. & Sugiyama, M.. (2014). Analysis of Empirical MAP and Empirical Partially Bayes: Can They be Alternatives to Variational Bayes?. Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 33:20-28 Available from https://proceedings.mlr.press/v33/nakajima14.html.

Related Material