Fairness interventions and case shift

Screening applications, including welfare fraud detection, credit scoring, and criminal risk assessment, are among the most high-stakes settings for algorithmic fairness. They share a structural property. Because outcome labels are available only for cases selected by a prior screening process, the training distribution differs fundamentally from the deployment population. Fairness interventions are calibrated on such training data but must hold for the population. Here, we analyze the implications of this mismatch for the effectiveness of fairness interventions and identify two failure modes. The first is the non-identifiability of fairness measures. Most standard fairness measures require confusion matrix entries that are structurally absent under selective labeling and cannot be estimated regardless of sample size. The second is disparity reversal. Selective labeling distorts class prevalence in the training corpus relative to the population. When the model is additionally differentially accurate across demographic groups, a fairness intervention that achieves parity at the training prevalence can reverse the disparity at deployment. We derive conditions under which reversal occurs and show that non-identifiability prevents its detection on available data. We validate both failure modes against a documented case of welfare fraud detection by the City of Amsterdam in which an extensive fairness intervention corrected for disparity on the evaluation corpus but reversed it in a live pilot. We recommend that only identified measures should guide fairness interventions, that differential accuracy should be minimized or explicitly accounted for, and that the prevalence mismatch between the corpus and the deployment population should be corrected before optimizing models for fairness.