Spread Divergence

Mingtian Zhang, Peter Hayes, Thomas Bird, Raza Habib, David Barber
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:11106-11116, 2020.

Abstract

For distributions $\mathbb{P}$ and $\mathbb{Q}$ with different supports or undefined densities, the divergence $\textrm{D}(\mathbb{P}||\mathbb{Q})$ may not exist. We define a Spread Divergence $\tilde{\textrm{D}}(\mathbb{P}||\mathbb{Q})$ on modified $\mathbb{P}$ and $\mathbb{Q}$ and describe sufficient conditions for the existence of such a divergence. We demonstrate how to maximize the discriminatory power of a given divergence by parameterizing and learning the spread. We also give examples of using a Spread Divergence to train implicit generative models, including linear models (Independent Components Analysis) and non-linear models (Deep Generative Networks).

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-zhang20j, title = {Spread Divergence}, author = {Zhang, Mingtian and Hayes, Peter and Bird, Thomas and Habib, Raza and Barber, David}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {11106--11116}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/zhang20j/zhang20j.pdf}, url = {https://proceedings.mlr.press/v119/zhang20j.html}, abstract = {For distributions $\mathbb{P}$ and $\mathbb{Q}$ with different supports or undefined densities, the divergence $\textrm{D}(\mathbb{P}||\mathbb{Q})$ may not exist. We define a Spread Divergence $\tilde{\textrm{D}}(\mathbb{P}||\mathbb{Q})$ on modified $\mathbb{P}$ and $\mathbb{Q}$ and describe sufficient conditions for the existence of such a divergence. We demonstrate how to maximize the discriminatory power of a given divergence by parameterizing and learning the spread. We also give examples of using a Spread Divergence to train implicit generative models, including linear models (Independent Components Analysis) and non-linear models (Deep Generative Networks).} }
Endnote
%0 Conference Paper %T Spread Divergence %A Mingtian Zhang %A Peter Hayes %A Thomas Bird %A Raza Habib %A David Barber %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-zhang20j %I PMLR %P 11106--11116 %U https://proceedings.mlr.press/v119/zhang20j.html %V 119 %X For distributions $\mathbb{P}$ and $\mathbb{Q}$ with different supports or undefined densities, the divergence $\textrm{D}(\mathbb{P}||\mathbb{Q})$ may not exist. We define a Spread Divergence $\tilde{\textrm{D}}(\mathbb{P}||\mathbb{Q})$ on modified $\mathbb{P}$ and $\mathbb{Q}$ and describe sufficient conditions for the existence of such a divergence. We demonstrate how to maximize the discriminatory power of a given divergence by parameterizing and learning the spread. We also give examples of using a Spread Divergence to train implicit generative models, including linear models (Independent Components Analysis) and non-linear models (Deep Generative Networks).
APA
Zhang, M., Hayes, P., Bird, T., Habib, R. & Barber, D.. (2020). Spread Divergence. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:11106-11116 Available from https://proceedings.mlr.press/v119/zhang20j.html.

Related Material