Nonparametric Estimation of Renyi Divergence and Friends

Akshay Krishnamurthy, Kirthevasan Kandasamy, Barnabas Poczos, Larry Wasserman
Proceedings of the 31st International Conference on Machine Learning, PMLR 32(2):919-927, 2014.

Abstract

We consider nonparametric estimation of L_2, Renyi-αand Tsallis-αdivergences between continuous distributions. Our approach is to construct estimators for particular integral functionals of two densities and translate them into divergence estimators. For the integral functionals, our estimators are based on corrections of a preliminary plug-in estimator. We show that these estimators achieve the parametric convergence rate of n^-1/2 when the densities’ smoothness, s, are both at least d/4 where d is the dimension. We also derive minimax lower bounds for this problem which confirm that s > d/4 is necessary to achieve the n^-1/2 rate of convergence. We validate our theoretical guarantees with a number of simulations.

Cite this Paper

BibTeX
@InProceedings{pmlr-v32-krishnamurthy14, title = {Nonparametric Estimation of Renyi Divergence and Friends}, author = {Krishnamurthy, Akshay and Kandasamy, Kirthevasan and Poczos, Barnabas and Wasserman, Larry}, booktitle = {Proceedings of the 31st International Conference on Machine Learning}, pages = {919--927}, year = {2014}, editor = {Xing, Eric P. and Jebara, Tony}, volume = {32}, number = {2}, series = {Proceedings of Machine Learning Research}, address = {Bejing, China}, month = {22--24 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v32/krishnamurthy14.pdf}, url = {https://proceedings.mlr.press/v32/krishnamurthy14.html}, abstract = {We consider nonparametric estimation of L_2, Renyi-αand Tsallis-αdivergences between continuous distributions. Our approach is to construct estimators for particular integral functionals of two densities and translate them into divergence estimators. For the integral functionals, our estimators are based on corrections of a preliminary plug-in estimator. We show that these estimators achieve the parametric convergence rate of n^-1/2 when the densities’ smoothness, s, are both at least d/4 where d is the dimension. We also derive minimax lower bounds for this problem which confirm that s > d/4 is necessary to achieve the n^-1/2 rate of convergence. We validate our theoretical guarantees with a number of simulations.} }
Endnote
%0 Conference Paper %T Nonparametric Estimation of Renyi Divergence and Friends %A Akshay Krishnamurthy %A Kirthevasan Kandasamy %A Barnabas Poczos %A Larry Wasserman %B Proceedings of the 31st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2014 %E Eric P. Xing %E Tony Jebara %F pmlr-v32-krishnamurthy14 %I PMLR %P 919--927 %U https://proceedings.mlr.press/v32/krishnamurthy14.html %V 32 %N 2 %X We consider nonparametric estimation of L_2, Renyi-αand Tsallis-αdivergences between continuous distributions. Our approach is to construct estimators for particular integral functionals of two densities and translate them into divergence estimators. For the integral functionals, our estimators are based on corrections of a preliminary plug-in estimator. We show that these estimators achieve the parametric convergence rate of n^-1/2 when the densities’ smoothness, s, are both at least d/4 where d is the dimension. We also derive minimax lower bounds for this problem which confirm that s > d/4 is necessary to achieve the n^-1/2 rate of convergence. We validate our theoretical guarantees with a number of simulations.
RIS
TY - CPAPER TI - Nonparametric Estimation of Renyi Divergence and Friends AU - Akshay Krishnamurthy AU - Kirthevasan Kandasamy AU - Barnabas Poczos AU - Larry Wasserman BT - Proceedings of the 31st International Conference on Machine Learning DA - 2014/06/18 ED - Eric P. Xing ED - Tony Jebara ID - pmlr-v32-krishnamurthy14 PB - PMLR DP - Proceedings of Machine Learning Research VL - 32 IS - 2 SP - 919 EP - 927 L1 - http://proceedings.mlr.press/v32/krishnamurthy14.pdf UR - https://proceedings.mlr.press/v32/krishnamurthy14.html AB - We consider nonparametric estimation of L_2, Renyi-αand Tsallis-αdivergences between continuous distributions. Our approach is to construct estimators for particular integral functionals of two densities and translate them into divergence estimators. For the integral functionals, our estimators are based on corrections of a preliminary plug-in estimator. We show that these estimators achieve the parametric convergence rate of n^-1/2 when the densities’ smoothness, s, are both at least d/4 where d is the dimension. We also derive minimax lower bounds for this problem which confirm that s > d/4 is necessary to achieve the n^-1/2 rate of convergence. We validate our theoretical guarantees with a number of simulations. ER -
APA
Krishnamurthy, A., Kandasamy, K., Poczos, B. & Wasserman, L.. (2014). Nonparametric Estimation of Renyi Divergence and Friends. Proceedings of the 31st International Conference on Machine Learning, in Proceedings of Machine Learning Research 32(2):919-927 Available from https://proceedings.mlr.press/v32/krishnamurthy14.html.

Related Material