Auditing Feature Importance Rankings in Nonstationary Data Streams

Feature importance explanations are often evaluated on fixed tabular data, although many deployed models operate on nonstationary streams in which the data distribution and relevant signal may change over time. In such settings, a ranking can appear stable while failing to respond to drift, or change for reasons unrelated to the label signal learned by the model. This paper proposes a protocol for evaluating feature importance rankings in data streams. The protocol combines a sanity gate based on label randomization, deletion tests for faithfulness, keep-top-k sufficiency, temporal rank stability, negative controls, and Drift–Explanation Alignment (DEA), a diagnostic that relates distributional drift to changes in feature rankings. We evaluate the protocol on controlled synthetic streams representing stable, abrupt, gradual, and correlated proxy conditions, as well as selected real datasets from electricity markets, insect monitoring, gas sensing, and industrial fault detection. Controls that are independent of the trained model fail the sanity gate, even when they are perfectly stable or responsive to drift. Rankings derived from trained models recover known signal features in controlled streams but show mixed reliability on real data. These findings support interpreting DEA and temporal stability only after establishing model dependence.