Fair Risk Control: A Generalized Framework for Calibrating Multi-group Fairness Risks

Lujing Zhang, Aaron Roth, Linjun Zhang
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:59783-59805, 2024.

Abstract

This paper introduces a framework for post-processing machine learning models so that their predictions satisfy multi-group fairness guarantees. Based on the celebrated notion of multicalibration, we introduce $(s,g,\alpha)-$GMC (Generalized Multi-Dimensional Multicalibration) for multi-dimensional mappings $s$, constraints $g$, and a pre-specified threshold level $\alpha$. We propose associated algorithms to achieve this notion in general settings. This framework is then applied to diverse scenarios encompassing different fairness concerns, including false negative rate control in image segmentation, prediction set conditional uncertainty quantification in hierarchical classification, and de-biased text generation in language models. We conduct numerical studies on several datasets and tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-zhang24be, title = {Fair Risk Control: A Generalized Framework for Calibrating Multi-group Fairness Risks}, author = {Zhang, Lujing and Roth, Aaron and Zhang, Linjun}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {59783--59805}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/zhang24be/zhang24be.pdf}, url = {https://proceedings.mlr.press/v235/zhang24be.html}, abstract = {This paper introduces a framework for post-processing machine learning models so that their predictions satisfy multi-group fairness guarantees. Based on the celebrated notion of multicalibration, we introduce $(s,g,\alpha)-$GMC (Generalized Multi-Dimensional Multicalibration) for multi-dimensional mappings $s$, constraints $g$, and a pre-specified threshold level $\alpha$. We propose associated algorithms to achieve this notion in general settings. This framework is then applied to diverse scenarios encompassing different fairness concerns, including false negative rate control in image segmentation, prediction set conditional uncertainty quantification in hierarchical classification, and de-biased text generation in language models. We conduct numerical studies on several datasets and tasks.} }
Endnote
%0 Conference Paper %T Fair Risk Control: A Generalized Framework for Calibrating Multi-group Fairness Risks %A Lujing Zhang %A Aaron Roth %A Linjun Zhang %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-zhang24be %I PMLR %P 59783--59805 %U https://proceedings.mlr.press/v235/zhang24be.html %V 235 %X This paper introduces a framework for post-processing machine learning models so that their predictions satisfy multi-group fairness guarantees. Based on the celebrated notion of multicalibration, we introduce $(s,g,\alpha)-$GMC (Generalized Multi-Dimensional Multicalibration) for multi-dimensional mappings $s$, constraints $g$, and a pre-specified threshold level $\alpha$. We propose associated algorithms to achieve this notion in general settings. This framework is then applied to diverse scenarios encompassing different fairness concerns, including false negative rate control in image segmentation, prediction set conditional uncertainty quantification in hierarchical classification, and de-biased text generation in language models. We conduct numerical studies on several datasets and tasks.
APA
Zhang, L., Roth, A. & Zhang, L.. (2024). Fair Risk Control: A Generalized Framework for Calibrating Multi-group Fairness Risks. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:59783-59805 Available from https://proceedings.mlr.press/v235/zhang24be.html.

Related Material