Pseudo-Calibration: Improving Predictive Uncertainty Estimation in Unsupervised Domain Adaptation

Dapeng Hu, Jian Liang, Xinchao Wang, Chuan-Sheng Foo
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:19304-19326, 2024.

Abstract

Unsupervised domain adaptation (UDA) has seen substantial efforts to improve model accuracy for an unlabeled target domain with the help of a labeled source domain. However, UDA models often exhibit poorly calibrated predictive uncertainty on target data, a problem that remains under-explored and poses risks in safety-critical UDA applications. The calibration problem in UDA is particularly challenging due to the absence of labeled target data and severe distribution shifts between domains. In this paper, we approach UDA calibration as a target-domain-specific unsupervised problem, different from mainstream solutions based on covariate shift. We introduce Pseudo-Calibration (PseudoCal), a novel post-hoc calibration framework. Our innovative use of inference-stage mixup synthesizes a labeled pseudo-target set capturing the structure of the real unlabeled target data. This turns the unsupervised calibration problem into a supervised one, easily solvable with temperature scaling. Extensive empirical evaluations across 5 diverse UDA scenarios involving 10 UDA methods consistently demonstrate the superior performance and versatility of PseudoCal over existing solutions.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-hu24i, title = {Pseudo-Calibration: Improving Predictive Uncertainty Estimation in Unsupervised Domain Adaptation}, author = {Hu, Dapeng and Liang, Jian and Wang, Xinchao and Foo, Chuan-Sheng}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {19304--19326}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/hu24i/hu24i.pdf}, url = {https://proceedings.mlr.press/v235/hu24i.html}, abstract = {Unsupervised domain adaptation (UDA) has seen substantial efforts to improve model accuracy for an unlabeled target domain with the help of a labeled source domain. However, UDA models often exhibit poorly calibrated predictive uncertainty on target data, a problem that remains under-explored and poses risks in safety-critical UDA applications. The calibration problem in UDA is particularly challenging due to the absence of labeled target data and severe distribution shifts between domains. In this paper, we approach UDA calibration as a target-domain-specific unsupervised problem, different from mainstream solutions based on covariate shift. We introduce Pseudo-Calibration (PseudoCal), a novel post-hoc calibration framework. Our innovative use of inference-stage mixup synthesizes a labeled pseudo-target set capturing the structure of the real unlabeled target data. This turns the unsupervised calibration problem into a supervised one, easily solvable with temperature scaling. Extensive empirical evaluations across 5 diverse UDA scenarios involving 10 UDA methods consistently demonstrate the superior performance and versatility of PseudoCal over existing solutions.} }
Endnote
%0 Conference Paper %T Pseudo-Calibration: Improving Predictive Uncertainty Estimation in Unsupervised Domain Adaptation %A Dapeng Hu %A Jian Liang %A Xinchao Wang %A Chuan-Sheng Foo %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-hu24i %I PMLR %P 19304--19326 %U https://proceedings.mlr.press/v235/hu24i.html %V 235 %X Unsupervised domain adaptation (UDA) has seen substantial efforts to improve model accuracy for an unlabeled target domain with the help of a labeled source domain. However, UDA models often exhibit poorly calibrated predictive uncertainty on target data, a problem that remains under-explored and poses risks in safety-critical UDA applications. The calibration problem in UDA is particularly challenging due to the absence of labeled target data and severe distribution shifts between domains. In this paper, we approach UDA calibration as a target-domain-specific unsupervised problem, different from mainstream solutions based on covariate shift. We introduce Pseudo-Calibration (PseudoCal), a novel post-hoc calibration framework. Our innovative use of inference-stage mixup synthesizes a labeled pseudo-target set capturing the structure of the real unlabeled target data. This turns the unsupervised calibration problem into a supervised one, easily solvable with temperature scaling. Extensive empirical evaluations across 5 diverse UDA scenarios involving 10 UDA methods consistently demonstrate the superior performance and versatility of PseudoCal over existing solutions.
APA
Hu, D., Liang, J., Wang, X. & Foo, C.. (2024). Pseudo-Calibration: Improving Predictive Uncertainty Estimation in Unsupervised Domain Adaptation. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:19304-19326 Available from https://proceedings.mlr.press/v235/hu24i.html.

Related Material