Multiple Source Adaptation and the Renyi Divergence

Yishay Mansour, Mehryar Mohri, Afshin Rostamizadeh
Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence, PMLR R7:375-382, 2009.

Abstract

This paper presents a novel theoretical study of the general problem of multiple source adaptation using the notion of Renyi divergence. Our results build on our previous work [12], but significantly broaden the scope of that work in several directions. We extend previous multiple source loss guarantees based on distribution weighted combinations to arbitrary target distributions P, not necessarily mixtures of the source distributions, analyze both known and unknown target distribution cases, and prove a lower bound. We further extend our bounds to deal with the case where the learner receives an approximate distribution for each source instead of the exact one, and show that similar loss guarantees can be achieved depending on the divergence between the approximate and true distributions. We also analyze the case where the labeling functions of the source domains are somewhat different. Finally, we report the results of experiments with both an artificial data set and a sentiment analysis task, showing the performance benefits of the distribution weighted combinations and the quality of our bounds based on the Renyi divergence.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR7-mansour09a, title = {Multiple Source Adaptation and the Renyi Divergence}, author = {Mansour, Yishay and Mohri, Mehryar and Rostamizadeh, Afshin}, booktitle = {Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence}, pages = {375--382}, year = {2009}, editor = {Bilmes, Jeff and Ng, Andrew Y.}, volume = {R7}, series = {Proceedings of Machine Learning Research}, month = {18--21 Jun}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r7/main/assets/mansour09a/mansour09a.pdf}, url = {https://proceedings.mlr.press/r7/mansour09a.html}, abstract = {This paper presents a novel theoretical study of the general problem of multiple source adaptation using the notion of Renyi divergence. Our results build on our previous work [12], but significantly broaden the scope of that work in several directions. We extend previous multiple source loss guarantees based on distribution weighted combinations to arbitrary target distributions P, not necessarily mixtures of the source distributions, analyze both known and unknown target distribution cases, and prove a lower bound. We further extend our bounds to deal with the case where the learner receives an approximate distribution for each source instead of the exact one, and show that similar loss guarantees can be achieved depending on the divergence between the approximate and true distributions. We also analyze the case where the labeling functions of the source domains are somewhat different. Finally, we report the results of experiments with both an artificial data set and a sentiment analysis task, showing the performance benefits of the distribution weighted combinations and the quality of our bounds based on the Renyi divergence.}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T Multiple Source Adaptation and the Renyi Divergence %A Yishay Mansour %A Mehryar Mohri %A Afshin Rostamizadeh %B Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2009 %E Jeff Bilmes %E Andrew Y. Ng %F pmlr-vR7-mansour09a %I PMLR %P 375--382 %U https://proceedings.mlr.press/r7/mansour09a.html %V R7 %X This paper presents a novel theoretical study of the general problem of multiple source adaptation using the notion of Renyi divergence. Our results build on our previous work [12], but significantly broaden the scope of that work in several directions. We extend previous multiple source loss guarantees based on distribution weighted combinations to arbitrary target distributions P, not necessarily mixtures of the source distributions, analyze both known and unknown target distribution cases, and prove a lower bound. We further extend our bounds to deal with the case where the learner receives an approximate distribution for each source instead of the exact one, and show that similar loss guarantees can be achieved depending on the divergence between the approximate and true distributions. We also analyze the case where the labeling functions of the source domains are somewhat different. Finally, we report the results of experiments with both an artificial data set and a sentiment analysis task, showing the performance benefits of the distribution weighted combinations and the quality of our bounds based on the Renyi divergence. %Z Reissued by PMLR on 04 October 2026.
APA
Mansour, Y., Mohri, M. & Rostamizadeh, A.. (2009). Multiple Source Adaptation and the Renyi Divergence. Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R7:375-382 Available from https://proceedings.mlr.press/r7/mansour09a.html. Reissued by PMLR on 04 October 2026.

Related Material