Calibration-Aware Online Adaptation under Label Shift

We study online adaptation of a pre-trained base classifier to streaming unlabeled data under label shift, where the marginal label proportions in the stream differ from those in the offline training data. We consider the case where the base classifier’s model class may be misspecified, motivating a separate, calibrated auxiliary classifier used solely to estimate the target label proportions. While many works have studied this setting, it is less understood how calibration quality affects the performance of the adapted base classifier in an online setting. In this paper, we thoroughly analyze the estimation error of the adapted base classifier after the deployment of a novel algorithm that estimates the target label proportions in an online fashion and dynamically adapts the base classifier using the estimated proportions. We decompose the error into a term that vanishes over time and a term determined by calibration quality. Moreover, we characterize an explicit trade-off between calibration granularity and finite-sample calibration error, and propose a novel calibration strategy which effectively balances this trade-off.