Beyond Bounds: Quantifying the Probability of Counterfactual Fairness

Taehan Kim, Minyoung Cho, Sanghack Lee
Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, PMLR 337:2994-3011, 2026.

Abstract

Counterfactual fairness is a rigorous criterion for algorithmic decision-making but remains fundamentally unidentifiable from observational data alone. Existing partial identification methods address this by deriving bounds for fairness measures; however, these intervals are often wide, limiting their practical utility. To address this limitation, we propose a framework that quantifies the probability that a black-box model satisfies counterfactual fairness, under a stated prior over the structural causal models compatible with the observed data. By exploiting conditional independencies to reduce the parameter space, we demonstrate that the set of causal parameters compatible with the observed data forms a convex polytope. We further show that, given domain-specific priors on exogenous distributions, this prior-dependent probability can be estimated via Hit-and-Run Monte Carlo integration. Our approach provides a practical tool that complements worst-case bounds, offering a prior-dependent probabilistic summary for auditing model fairness.

Cite this Paper

BibTeX
@InProceedings{pmlr-v337-kim26b, title = {Beyond Bounds: Quantifying the Probability of Counterfactual Fairness}, author = {Kim, Taehan and Cho, Minyoung and Lee, Sanghack}, booktitle = {Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence}, pages = {2994--3011}, year = {2026}, editor = {Perković, Emilija and Malinsky, Daniel}, volume = {337}, series = {Proceedings of Machine Learning Research}, month = {17--21 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v337/main/assets/kim26b/kim26b.pdf}, url = {https://proceedings.mlr.press/v337/kim26b.html}, abstract = {Counterfactual fairness is a rigorous criterion for algorithmic decision-making but remains fundamentally unidentifiable from observational data alone. Existing partial identification methods address this by deriving bounds for fairness measures; however, these intervals are often wide, limiting their practical utility. To address this limitation, we propose a framework that quantifies the probability that a black-box model satisfies counterfactual fairness, under a stated prior over the structural causal models compatible with the observed data. By exploiting conditional independencies to reduce the parameter space, we demonstrate that the set of causal parameters compatible with the observed data forms a convex polytope. We further show that, given domain-specific priors on exogenous distributions, this prior-dependent probability can be estimated via Hit-and-Run Monte Carlo integration. Our approach provides a practical tool that complements worst-case bounds, offering a prior-dependent probabilistic summary for auditing model fairness.} }
Endnote
%0 Conference Paper %T Beyond Bounds: Quantifying the Probability of Counterfactual Fairness %A Taehan Kim %A Minyoung Cho %A Sanghack Lee %B Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2026 %E Emilija Perković %E Daniel Malinsky %F pmlr-v337-kim26b %I PMLR %P 2994--3011 %U https://proceedings.mlr.press/v337/kim26b.html %V 337 %X Counterfactual fairness is a rigorous criterion for algorithmic decision-making but remains fundamentally unidentifiable from observational data alone. Existing partial identification methods address this by deriving bounds for fairness measures; however, these intervals are often wide, limiting their practical utility. To address this limitation, we propose a framework that quantifies the probability that a black-box model satisfies counterfactual fairness, under a stated prior over the structural causal models compatible with the observed data. By exploiting conditional independencies to reduce the parameter space, we demonstrate that the set of causal parameters compatible with the observed data forms a convex polytope. We further show that, given domain-specific priors on exogenous distributions, this prior-dependent probability can be estimated via Hit-and-Run Monte Carlo integration. Our approach provides a practical tool that complements worst-case bounds, offering a prior-dependent probabilistic summary for auditing model fairness.
APA
Kim, T., Cho, M. & Lee, S.. (2026). Beyond Bounds: Quantifying the Probability of Counterfactual Fairness. Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 337:2994-3011 Available from https://proceedings.mlr.press/v337/kim26b.html.

Related Material