Estimating Mutual Information by Local Gaussian Approximation

Shuyang Gao USC, Greg Ver Steeg Information Sciences Institute, Aram Galstyan Information Sciences Institute
Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence, PMLR R13:664-671, 2015.

Abstract

A common problem found in machine learning, data analysis, and statistics is the estimation of mutual information. Previous works have shown that non-parametric estimation of mutual information is more difficult for strongly dependent variables. We present Local Gaussian Approximation (LGA), a simple semi-parametric estimator of mutual information based on finite i.i.d. samples drawn from an unknown probability distribution. We estimate mutual information as a sample expectation of log density ratios. At each sample point, densities are locally approximated via Gaussians. We show the consistency of our method and demonstrate that, unlike existing methods, the new estimator is able to accurately measure relationship strengths over many orders of magnitude.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR13-usc15a, title = {Estimating Mutual Information by Local {G}aussian Approximation}, author = {USC, Shuyang Gao and Institute, Greg Ver Steeg Information Sciences and Institute, Aram Galstyan Information Sciences}, booktitle = {Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence}, pages = {664--671}, year = {2015}, editor = {Meila, Marina and Heskes, Tom}, volume = {R13}, series = {Proceedings of Machine Learning Research}, month = {12--16 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r13/main/assets/usc15a/usc15a.pdf}, url = {https://proceedings.mlr.press/r13/usc15a.html}, abstract = {A common problem found in machine learning, data analysis, and statistics is the estimation of mutual information. Previous works have shown that non-parametric estimation of mutual information is more difficult for strongly dependent variables. We present Local Gaussian Approximation (LGA), a simple semi-parametric estimator of mutual information based on finite i.i.d. samples drawn from an unknown probability distribution. We estimate mutual information as a sample expectation of log density ratios. At each sample point, densities are locally approximated via Gaussians. We show the consistency of our method and demonstrate that, unlike existing methods, the new estimator is able to accurately measure relationship strengths over many orders of magnitude.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Estimating Mutual Information by Local Gaussian Approximation %A Shuyang Gao USC %A Greg Ver Steeg Information Sciences Institute %A Aram Galstyan Information Sciences Institute %B Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2015 %E Marina Meila %E Tom Heskes %F pmlr-vR13-usc15a %I PMLR %P 664--671 %U https://proceedings.mlr.press/r13/usc15a.html %V R13 %X A common problem found in machine learning, data analysis, and statistics is the estimation of mutual information. Previous works have shown that non-parametric estimation of mutual information is more difficult for strongly dependent variables. We present Local Gaussian Approximation (LGA), a simple semi-parametric estimator of mutual information based on finite i.i.d. samples drawn from an unknown probability distribution. We estimate mutual information as a sample expectation of log density ratios. At each sample point, densities are locally approximated via Gaussians. We show the consistency of our method and demonstrate that, unlike existing methods, the new estimator is able to accurately measure relationship strengths over many orders of magnitude. %Z Reissued by PMLR on 04 October 2026.
APA
USC, S.G., Institute, G.V.S.I.S. & Institute, A.G.I.S.. (2015). Estimating Mutual Information by Local Gaussian Approximation. Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R13:664-671 Available from https://proceedings.mlr.press/r13/usc15a.html. Reissued by PMLR on 04 October 2026.

Related Material