Exposing Hidden Vulnerabilities: A privacy audit of algorithmic fairness

Niloy Purkait, Henry Brighton, Emmanuel Keuleers
Proceedings of Fourth European Workshop on Algorithmic Fairness, PMLR 294:232-250, 2025.

Abstract

Algorithmic fairness and privacy are cornerstones of trustworthy machine learning. Prior research suggests that mitigating bias may amplify membership inference risks for underrepresented groups. However, evaluations based on average-case metrics, such as attack success rate, often obscure vulnerabilities at low false-positive rates-an regime critical for real-world privacy. In this paper, we present an in-depth empirical study of membership inference under low false positive rate conditions, across three fair machine learning algorithms. Our analysis on the Law School, Bank Marketing, and COMPAS datasets reveal that standard threshold-based and population-based attacks underestimate privacy leaks, especially for sensitive subgroups. Motivated by this gap, we propose a subgroup-specific membership inference attack based on pairwise likelihood ratio tests. Our method models the null hypothesis more accurately: a target sample’s outputs are statistically indistinguishable from those of non-members within the same sensitive subgroup. Our experiments show that a simple modification of an existing attack can achieve superior test power across the TPR-FPR curve-even at extremely low false positive rates, given an adversary with the same computational resources and access to data-establishing a robust foundation for more powerful, fine-grained privacy audits of fair learning algorithms.

Cite this Paper

BibTeX
@InProceedings{pmlr-v294-purkait25a, title = {Exposing Hidden Vulnerabilities: A privacy audit of algorithmic fairness}, author = {Purkait, Niloy and Brighton, Henry and Keuleers, Emmanuel}, booktitle = {Proceedings of Fourth European Workshop on Algorithmic Fairness}, pages = {232--250}, year = {2025}, editor = {Weerts, Hilde and Pechenizkiy, Mykola and Allhutter, Doris and Corrêa, Ana Maria and Grote, Thomas and Liem, Cynthia}, volume = {294}, series = {Proceedings of Machine Learning Research}, month = {30 Jun--02 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v294/main/assets/purkait25a/purkait25a.pdf}, url = {https://proceedings.mlr.press/v294/purkait25a.html}, abstract = {Algorithmic fairness and privacy are cornerstones of trustworthy machine learning. Prior research suggests that mitigating bias may amplify membership inference risks for underrepresented groups. However, evaluations based on average-case metrics, such as attack success rate, often obscure vulnerabilities at low false-positive rates-an regime critical for real-world privacy. In this paper, we present an in-depth empirical study of membership inference under low false positive rate conditions, across three fair machine learning algorithms. Our analysis on the Law School, Bank Marketing, and COMPAS datasets reveal that standard threshold-based and population-based attacks underestimate privacy leaks, especially for sensitive subgroups. Motivated by this gap, we propose a subgroup-specific membership inference attack based on pairwise likelihood ratio tests. Our method models the null hypothesis more accurately: a target sample’s outputs are statistically indistinguishable from those of non-members within the same sensitive subgroup. Our experiments show that a simple modification of an existing attack can achieve superior test power across the TPR-FPR curve-even at extremely low false positive rates, given an adversary with the same computational resources and access to data-establishing a robust foundation for more powerful, fine-grained privacy audits of fair learning algorithms.} }
Endnote
%0 Conference Paper %T Exposing Hidden Vulnerabilities: A privacy audit of algorithmic fairness %A Niloy Purkait %A Henry Brighton %A Emmanuel Keuleers %B Proceedings of Fourth European Workshop on Algorithmic Fairness %C Proceedings of Machine Learning Research %D 2025 %E Hilde Weerts %E Mykola Pechenizkiy %E Doris Allhutter %E Ana Maria Corrêa %E Thomas Grote %E Cynthia Liem %F pmlr-v294-purkait25a %I PMLR %P 232--250 %U https://proceedings.mlr.press/v294/purkait25a.html %V 294 %X Algorithmic fairness and privacy are cornerstones of trustworthy machine learning. Prior research suggests that mitigating bias may amplify membership inference risks for underrepresented groups. However, evaluations based on average-case metrics, such as attack success rate, often obscure vulnerabilities at low false-positive rates-an regime critical for real-world privacy. In this paper, we present an in-depth empirical study of membership inference under low false positive rate conditions, across three fair machine learning algorithms. Our analysis on the Law School, Bank Marketing, and COMPAS datasets reveal that standard threshold-based and population-based attacks underestimate privacy leaks, especially for sensitive subgroups. Motivated by this gap, we propose a subgroup-specific membership inference attack based on pairwise likelihood ratio tests. Our method models the null hypothesis more accurately: a target sample’s outputs are statistically indistinguishable from those of non-members within the same sensitive subgroup. Our experiments show that a simple modification of an existing attack can achieve superior test power across the TPR-FPR curve-even at extremely low false positive rates, given an adversary with the same computational resources and access to data-establishing a robust foundation for more powerful, fine-grained privacy audits of fair learning algorithms.
APA
Purkait, N., Brighton, H. & Keuleers, E.. (2025). Exposing Hidden Vulnerabilities: A privacy audit of algorithmic fairness. Proceedings of Fourth European Workshop on Algorithmic Fairness, in Proceedings of Machine Learning Research 294:232-250 Available from https://proceedings.mlr.press/v294/purkait25a.html.

Related Material