Output-Distribution Divergence as a Pre-Interpretation Gate for Mental Health AI

Machine learning models in mental health are widely used to generate risk scores from observational data, yet their outputs are frequently interpreted in causal or intervention-oriented terms without explicit checks on whether such interpretations transport across demographic contexts. We propose a simple governance-oriented diagnostic that compares predicted-probability distributions across contexts against within-context baselines using standard divergence measures, functioning as a pre-interpretation gate rather than a causal estimator. We operationalize this protocol using Jensen–Shannon divergence and Wasserstein distance, calibrated via bootstrapped intra-context baselines, and evaluate it on depression risk prediction using PHQ-9 data from the National Health and Nutrition Examination Survey (NHANES) 2017–2020. Across age and sex contexts, we find that cross-context divergence consistently exceeds baseline variation, particularly for age-group transfers, even when discrimination metrics such as AUC remain stable. These results demonstrate that performance-based validation alone can mask substantial distributional instability in predicted probabilities, with implications for calibration and interpretability. We argue that output-distribution divergence provides a low-cost, model-agnostic diagnostic for identifying transportability risk prior to deploying or interpreting mental health prediction models in intervention-relevant settings.