Minimizing $f$-Divergences by Interpolating Velocity Fields

Song Liu, Jiahao Yu, Jack Simons, Mingxuan Yi, Mark Beaumont
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:32308-32331, 2024.

Abstract

Many machine learning problems can be seen as approximating a target distribution using a particle distribution by minimizing their statistical discrepancy. Wasserstein Gradient Flow can move particles along a path that minimizes the $f$-divergence between the target and particle distributions. To move particles, we need to calculate the corresponding velocity fields derived from a density ratio function between these two distributions. Previous works estimated such density ratio functions and then differentiated the estimated ratios. These approaches may suffer from overfitting, leading to a less accurate estimate of the velocity fields. Inspired by non-parametric curve fitting, we directly estimate these velocity fields using interpolation techniques. We prove that our estimators are consistent under mild conditions. We validate their effectiveness using novel applications on domain adaptation and missing data imputation. The code for reproducing our results can be found at https://github.com/anewgithubname/gradest2.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-liu24by, title = {Minimizing $f$-Divergences by Interpolating Velocity Fields}, author = {Liu, Song and Yu, Jiahao and Simons, Jack and Yi, Mingxuan and Beaumont, Mark}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {32308--32331}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/liu24by/liu24by.pdf}, url = {https://proceedings.mlr.press/v235/liu24by.html}, abstract = {Many machine learning problems can be seen as approximating a target distribution using a particle distribution by minimizing their statistical discrepancy. Wasserstein Gradient Flow can move particles along a path that minimizes the $f$-divergence between the target and particle distributions. To move particles, we need to calculate the corresponding velocity fields derived from a density ratio function between these two distributions. Previous works estimated such density ratio functions and then differentiated the estimated ratios. These approaches may suffer from overfitting, leading to a less accurate estimate of the velocity fields. Inspired by non-parametric curve fitting, we directly estimate these velocity fields using interpolation techniques. We prove that our estimators are consistent under mild conditions. We validate their effectiveness using novel applications on domain adaptation and missing data imputation. The code for reproducing our results can be found at https://github.com/anewgithubname/gradest2.} }
Endnote
%0 Conference Paper %T Minimizing $f$-Divergences by Interpolating Velocity Fields %A Song Liu %A Jiahao Yu %A Jack Simons %A Mingxuan Yi %A Mark Beaumont %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-liu24by %I PMLR %P 32308--32331 %U https://proceedings.mlr.press/v235/liu24by.html %V 235 %X Many machine learning problems can be seen as approximating a target distribution using a particle distribution by minimizing their statistical discrepancy. Wasserstein Gradient Flow can move particles along a path that minimizes the $f$-divergence between the target and particle distributions. To move particles, we need to calculate the corresponding velocity fields derived from a density ratio function between these two distributions. Previous works estimated such density ratio functions and then differentiated the estimated ratios. These approaches may suffer from overfitting, leading to a less accurate estimate of the velocity fields. Inspired by non-parametric curve fitting, we directly estimate these velocity fields using interpolation techniques. We prove that our estimators are consistent under mild conditions. We validate their effectiveness using novel applications on domain adaptation and missing data imputation. The code for reproducing our results can be found at https://github.com/anewgithubname/gradest2.
APA
Liu, S., Yu, J., Simons, J., Yi, M. & Beaumont, M.. (2024). Minimizing $f$-Divergences by Interpolating Velocity Fields. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:32308-32331 Available from https://proceedings.mlr.press/v235/liu24by.html.

Related Material