CID: Measuring Feature Importance Through Counterfactual Distributions

Eddie Conti, Álvaro Parafita, Axel Brando
Proceedings of the 7th Northern Lights Deep Learning Conference (NLDL), PMLR 307:72-85, 2026.

Abstract

Assessing the importance of individual features in Machine Learning is critical to understand the model’s decision-making process. While numerous methods exist, the lack of a definitive ground truth for comparison highlights the need for alternative, well-founded measures. This paper introduces a novel post-hoc local feature importance method called Counterfactual Importance Distribution (CID). We generate two sets of positive and negative counterfactuals, model their distributions using Kernel Density Estimation, and rank features based on a distributional dissimilarity measure. This measure, grounded in a rigorous mathematical framework, satisfies key properties required to function as a valid metric. We showcase the effectiveness of our method by comparing with well-established local feature importance explainers. Our method not only offers complementary perspectives to existing approaches, but also improves performance on faithfulness metrics (both for comprehensiveness and sufficiency), resulting in more faithful explanations of the system. These results highlight its potential as a valuable tool for model analysis.

Cite this Paper

BibTeX
@InProceedings{pmlr-v307-conti26a, title = {{CID}: Measuring Feature Importance Through Counterfactual Distributions}, author = {Conti, Eddie and Parafita, {\'A}lvaro and Brando, Axel}, booktitle = {Proceedings of the 7th Northern Lights Deep Learning Conference (NLDL)}, pages = {72--85}, year = {2026}, editor = {Kim, Hyeongji and Ramírez Rivera, Adín and Ricaud, Benjamin}, volume = {307}, series = {Proceedings of Machine Learning Research}, month = {06--08 Jan}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v307/main/assets/conti26a/conti26a.pdf}, url = {https://proceedings.mlr.press/v307/conti26a.html}, abstract = {Assessing the importance of individual features in Machine Learning is critical to understand the model’s decision-making process. While numerous methods exist, the lack of a definitive ground truth for comparison highlights the need for alternative, well-founded measures. This paper introduces a novel post-hoc local feature importance method called Counterfactual Importance Distribution (CID). We generate two sets of positive and negative counterfactuals, model their distributions using Kernel Density Estimation, and rank features based on a distributional dissimilarity measure. This measure, grounded in a rigorous mathematical framework, satisfies key properties required to function as a valid metric. We showcase the effectiveness of our method by comparing with well-established local feature importance explainers. Our method not only offers complementary perspectives to existing approaches, but also improves performance on faithfulness metrics (both for comprehensiveness and sufficiency), resulting in more faithful explanations of the system. These results highlight its potential as a valuable tool for model analysis.} }
Endnote
%0 Conference Paper %T CID: Measuring Feature Importance Through Counterfactual Distributions %A Eddie Conti %A Álvaro Parafita %A Axel Brando %B Proceedings of the 7th Northern Lights Deep Learning Conference (NLDL) %C Proceedings of Machine Learning Research %D 2026 %E Hyeongji Kim %E Adín Ramírez Rivera %E Benjamin Ricaud %F pmlr-v307-conti26a %I PMLR %P 72--85 %U https://proceedings.mlr.press/v307/conti26a.html %V 307 %X Assessing the importance of individual features in Machine Learning is critical to understand the model’s decision-making process. While numerous methods exist, the lack of a definitive ground truth for comparison highlights the need for alternative, well-founded measures. This paper introduces a novel post-hoc local feature importance method called Counterfactual Importance Distribution (CID). We generate two sets of positive and negative counterfactuals, model their distributions using Kernel Density Estimation, and rank features based on a distributional dissimilarity measure. This measure, grounded in a rigorous mathematical framework, satisfies key properties required to function as a valid metric. We showcase the effectiveness of our method by comparing with well-established local feature importance explainers. Our method not only offers complementary perspectives to existing approaches, but also improves performance on faithfulness metrics (both for comprehensiveness and sufficiency), resulting in more faithful explanations of the system. These results highlight its potential as a valuable tool for model analysis.
APA
Conti, E., Parafita, Á. & Brando, A.. (2026). CID: Measuring Feature Importance Through Counterfactual Distributions. Proceedings of the 7th Northern Lights Deep Learning Conference (NLDL), in Proceedings of Machine Learning Research 307:72-85 Available from https://proceedings.mlr.press/v307/conti26a.html.

Related Material