MMG: Mutual Information Estimation via the MMSE Gap in Diffusion

Mutual information is one of the most general ways to measure relationships between random variables, but estimating this quantity for complex systems is challenging. Denoising diffusion models have recently set a new bar for density estimation, so it is natural to consider whether these methods could also be used to improve mutual information estimation. Using the recently introduced information-theoretic formulation of denoising diffusion models, we show that diffusion models can be used in a straightforward way to estimate mutual information. In particular, the mutual information corresponds to half the gap in the minimum mean square error between conditional and unconditional diffusion, integrated over all signal-to-noise ratios in the noising process. Our approach not only passes self-consistency tests but also outperforms traditional and score-based diffusion estimators. Furthermore, our method leverages adaptive importance sampling to achieve scalable estimation, while maintaining strong performance even when the mutual information is high. Code to reproduce our experiments is available at \url{https://github.com/fengsxy/Diffusion_MI}, and the unified mutual information estimation library is available at \url{https://github.com/fengsxy/Diffusion-MI}.