Toward Calibrated Mixture-of-Experts Under Distribution Shift

Gina Wong, Drew Prinster, Suchi Saria, Rama Chellappa, Anqi Liu
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:135371-135400, 2026.

Abstract

Calibration aligns a model’s predictive uncertainty with the frequencies of its empirical outcomes and is important for understanding and trusting reported probabilities. Recent work shows that enforcing calibration at the level of individual predictors can improve ensemble accuracy and calibration, with mixture-of-experts (MoE) models showing strong empirical improvements in particular; however, the conditions under which calibration helps MoE are not well understood. In this work, we study how MoE models behave under distribution shift, focusing on how routing mechanisms interact with expert-level calibration. We show that expert calibration is sufficient to ensure calibration of the overall model under a broad class of distribution shifts in hard-routed models, but is insufficient for calibrating soft-routed models. To address this, we propose an adversarial reweighting that penalizes calibration errors of the routed aggregate under distribution shift, and we demonstrate that it improves the accuracy-calibration tradeoff both on average and on difficult subsets of the data, across model classes, prediction tasks, and distribution shifts.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-wong26a, title = {Toward Calibrated Mixture-of-Experts Under Distribution Shift}, author = {Wong, Gina and Prinster, Drew and Saria, Suchi and Chellappa, Rama and Liu, Anqi}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {135371--135400}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/wong26a/wong26a.pdf}, url = {https://proceedings.mlr.press/v306/wong26a.html}, abstract = {Calibration aligns a model’s predictive uncertainty with the frequencies of its empirical outcomes and is important for understanding and trusting reported probabilities. Recent work shows that enforcing calibration at the level of individual predictors can improve ensemble accuracy and calibration, with mixture-of-experts (MoE) models showing strong empirical improvements in particular; however, the conditions under which calibration helps MoE are not well understood. In this work, we study how MoE models behave under distribution shift, focusing on how routing mechanisms interact with expert-level calibration. We show that expert calibration is sufficient to ensure calibration of the overall model under a broad class of distribution shifts in hard-routed models, but is insufficient for calibrating soft-routed models. To address this, we propose an adversarial reweighting that penalizes calibration errors of the routed aggregate under distribution shift, and we demonstrate that it improves the accuracy-calibration tradeoff both on average and on difficult subsets of the data, across model classes, prediction tasks, and distribution shifts.} }
Endnote
%0 Conference Paper %T Toward Calibrated Mixture-of-Experts Under Distribution Shift %A Gina Wong %A Drew Prinster %A Suchi Saria %A Rama Chellappa %A Anqi Liu %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-wong26a %I PMLR %P 135371--135400 %U https://proceedings.mlr.press/v306/wong26a.html %V 306 %X Calibration aligns a model’s predictive uncertainty with the frequencies of its empirical outcomes and is important for understanding and trusting reported probabilities. Recent work shows that enforcing calibration at the level of individual predictors can improve ensemble accuracy and calibration, with mixture-of-experts (MoE) models showing strong empirical improvements in particular; however, the conditions under which calibration helps MoE are not well understood. In this work, we study how MoE models behave under distribution shift, focusing on how routing mechanisms interact with expert-level calibration. We show that expert calibration is sufficient to ensure calibration of the overall model under a broad class of distribution shifts in hard-routed models, but is insufficient for calibrating soft-routed models. To address this, we propose an adversarial reweighting that penalizes calibration errors of the routed aggregate under distribution shift, and we demonstrate that it improves the accuracy-calibration tradeoff both on average and on difficult subsets of the data, across model classes, prediction tasks, and distribution shifts.
APA
Wong, G., Prinster, D., Saria, S., Chellappa, R. & Liu, A.. (2026). Toward Calibrated Mixture-of-Experts Under Distribution Shift. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:135371-135400 Available from https://proceedings.mlr.press/v306/wong26a.html.

Related Material