A Benchmark for Client-level Fairness in Federated Learning

Xenia Heilmann, Luca Corbucci, Mattia Cerrato
Proceedings of Fourth European Workshop on Algorithmic Fairness, PMLR 294:434-438, 2025.

Abstract

Federated Learning (FL) enables collaborative model training while preserving participating clients’ local data privacy. However, the diverse data distributions across different clients can exacerbate fairness issues, as biases in client data may propagate across the Federation. Although various approaches have been proposed to enhance fairness in FL, they typically focus on mitigating the bias of a single binary-sensitive attribute. This narrow focus often overlooks the complexity introduced by clients with conflicting or diverse fairness objectives. Such clients may contribute to the Federation without experiencing any improvement in their model’s performance or fairness regarding their specific sensitive attributes. To evaluate Fair FL methods for global and individual client fairness in a reproducible and reliable manner, the need for standardized datasets becomes apparent. In this paper, we propose a preliminary framework to create benchmarking datasets that allow researchers and developers to evaluate Fair FL methods. These benchmarking datasets include various heterogeneous client settings with regard to data bias of sensitive attributes for assessing fairness at the global and individual level. Additionally, we provide information on how to evaluate results obtained with these benchmarking datasets.

Cite this Paper

BibTeX
@InProceedings{pmlr-v294-heilmann25a, title = {A Benchmark for Client-level Fairness in Federated Learning}, author = {Heilmann, Xenia and Corbucci, Luca and Cerrato, Mattia}, booktitle = {Proceedings of Fourth European Workshop on Algorithmic Fairness}, pages = {434--438}, year = {2025}, editor = {Weerts, Hilde and Pechenizkiy, Mykola and Allhutter, Doris and Corrêa, Ana Maria and Grote, Thomas and Liem, Cynthia}, volume = {294}, series = {Proceedings of Machine Learning Research}, month = {30 Jun--02 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v294/main/assets/heilmann25a/heilmann25a.pdf}, url = {https://proceedings.mlr.press/v294/heilmann25a.html}, abstract = {Federated Learning (FL) enables collaborative model training while preserving participating clients’ local data privacy. However, the diverse data distributions across different clients can exacerbate fairness issues, as biases in client data may propagate across the Federation. Although various approaches have been proposed to enhance fairness in FL, they typically focus on mitigating the bias of a single binary-sensitive attribute. This narrow focus often overlooks the complexity introduced by clients with conflicting or diverse fairness objectives. Such clients may contribute to the Federation without experiencing any improvement in their model’s performance or fairness regarding their specific sensitive attributes. To evaluate Fair FL methods for global and individual client fairness in a reproducible and reliable manner, the need for standardized datasets becomes apparent. In this paper, we propose a preliminary framework to create benchmarking datasets that allow researchers and developers to evaluate Fair FL methods. These benchmarking datasets include various heterogeneous client settings with regard to data bias of sensitive attributes for assessing fairness at the global and individual level. Additionally, we provide information on how to evaluate results obtained with these benchmarking datasets.} }
Endnote
%0 Conference Paper %T A Benchmark for Client-level Fairness in Federated Learning %A Xenia Heilmann %A Luca Corbucci %A Mattia Cerrato %B Proceedings of Fourth European Workshop on Algorithmic Fairness %C Proceedings of Machine Learning Research %D 2025 %E Hilde Weerts %E Mykola Pechenizkiy %E Doris Allhutter %E Ana Maria Corrêa %E Thomas Grote %E Cynthia Liem %F pmlr-v294-heilmann25a %I PMLR %P 434--438 %U https://proceedings.mlr.press/v294/heilmann25a.html %V 294 %X Federated Learning (FL) enables collaborative model training while preserving participating clients’ local data privacy. However, the diverse data distributions across different clients can exacerbate fairness issues, as biases in client data may propagate across the Federation. Although various approaches have been proposed to enhance fairness in FL, they typically focus on mitigating the bias of a single binary-sensitive attribute. This narrow focus often overlooks the complexity introduced by clients with conflicting or diverse fairness objectives. Such clients may contribute to the Federation without experiencing any improvement in their model’s performance or fairness regarding their specific sensitive attributes. To evaluate Fair FL methods for global and individual client fairness in a reproducible and reliable manner, the need for standardized datasets becomes apparent. In this paper, we propose a preliminary framework to create benchmarking datasets that allow researchers and developers to evaluate Fair FL methods. These benchmarking datasets include various heterogeneous client settings with regard to data bias of sensitive attributes for assessing fairness at the global and individual level. Additionally, we provide information on how to evaluate results obtained with these benchmarking datasets.
APA
Heilmann, X., Corbucci, L. & Cerrato, M.. (2025). A Benchmark for Client-level Fairness in Federated Learning. Proceedings of Fourth European Workshop on Algorithmic Fairness, in Proceedings of Machine Learning Research 294:434-438 Available from https://proceedings.mlr.press/v294/heilmann25a.html.

Related Material