Conformal Online Model Aggregation

Matteo Gasparin, Aaditya Ramdas
Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, PMLR 337:1663-1683, 2026.

Abstract

Conformal prediction equips machine learning models with a reasonable notion of uncertainty quantification under mild assumptions. However, conformal prediction only works if we fix the underlying model in advance. A relatively unaddressed issue in conformal prediction is that of model selection and/or aggregation: given a set of prediction models, which one should we conformalize? We suggest that instead of performing model selection, it can be prudent and practical to perform conformal set aggregation in an online, adaptive fashion. We propose a wrapper that takes in several conformal prediction sets, and outputs a single adaptively-combined prediction set. Our method, called conformal online model aggregation ({COMA}), is based on combining the prediction sets from several algorithms by weighted voting, and can be thought of as a sort of online stacking of the underlying conformal sets. As long as the input sets have (distribution-free) coverage guarantees, {COMA} retains coverage guarantees, under a negative correlation assumption between errors and weights. {COMA} is well suited to decentralized settings, where users are only willing to share their prediction sets for a new test point in a black-box fashion. As we demonstrate, it is also well suited to settings with distribution drift, where model selection is not advisable.

Cite this Paper

BibTeX
@InProceedings{pmlr-v337-gasparin26a, title = {Conformal Online Model Aggregation}, author = {Gasparin, Matteo and Ramdas, Aaditya}, booktitle = {Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence}, pages = {1663--1683}, year = {2026}, editor = {Perković, Emilija and Malinsky, Daniel}, volume = {337}, series = {Proceedings of Machine Learning Research}, month = {17--21 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v337/main/assets/gasparin26a/gasparin26a.pdf}, url = {https://proceedings.mlr.press/v337/gasparin26a.html}, abstract = {Conformal prediction equips machine learning models with a reasonable notion of uncertainty quantification under mild assumptions. However, conformal prediction only works if we fix the underlying model in advance. A relatively unaddressed issue in conformal prediction is that of model selection and/or aggregation: given a set of prediction models, which one should we conformalize? We suggest that instead of performing model selection, it can be prudent and practical to perform conformal set aggregation in an online, adaptive fashion. We propose a wrapper that takes in several conformal prediction sets, and outputs a single adaptively-combined prediction set. Our method, called conformal online model aggregation ({COMA}), is based on combining the prediction sets from several algorithms by weighted voting, and can be thought of as a sort of online stacking of the underlying conformal sets. As long as the input sets have (distribution-free) coverage guarantees, {COMA} retains coverage guarantees, under a negative correlation assumption between errors and weights. {COMA} is well suited to decentralized settings, where users are only willing to share their prediction sets for a new test point in a black-box fashion. As we demonstrate, it is also well suited to settings with distribution drift, where model selection is not advisable.} }
Endnote
%0 Conference Paper %T Conformal Online Model Aggregation %A Matteo Gasparin %A Aaditya Ramdas %B Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2026 %E Emilija Perković %E Daniel Malinsky %F pmlr-v337-gasparin26a %I PMLR %P 1663--1683 %U https://proceedings.mlr.press/v337/gasparin26a.html %V 337 %X Conformal prediction equips machine learning models with a reasonable notion of uncertainty quantification under mild assumptions. However, conformal prediction only works if we fix the underlying model in advance. A relatively unaddressed issue in conformal prediction is that of model selection and/or aggregation: given a set of prediction models, which one should we conformalize? We suggest that instead of performing model selection, it can be prudent and practical to perform conformal set aggregation in an online, adaptive fashion. We propose a wrapper that takes in several conformal prediction sets, and outputs a single adaptively-combined prediction set. Our method, called conformal online model aggregation ({COMA}), is based on combining the prediction sets from several algorithms by weighted voting, and can be thought of as a sort of online stacking of the underlying conformal sets. As long as the input sets have (distribution-free) coverage guarantees, {COMA} retains coverage guarantees, under a negative correlation assumption between errors and weights. {COMA} is well suited to decentralized settings, where users are only willing to share their prediction sets for a new test point in a black-box fashion. As we demonstrate, it is also well suited to settings with distribution drift, where model selection is not advisable.
APA
Gasparin, M. & Ramdas, A.. (2026). Conformal Online Model Aggregation. Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 337:1663-1683 Available from https://proceedings.mlr.press/v337/gasparin26a.html.

Related Material