Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated Learning

Gwen Legate, Lucas Caccia, Eugene Belilovsky
Proceedings of The 2nd Conference on Lifelong Learning Agents, PMLR 232:764-780, 2023.

Abstract

In Federated Learning a global model is learned by aggregating model updates computed at a set of independent client nodes. To reduce communication costs, multiple gradient steps are performed at each node prior to aggregation. A key challenge in this setting is data heterogeneity across clients resulting in differing local objectives. This can lead clients to overly minimize their own local objective consequently diverging from the global solution. We demonstrate that individual client models experience a catastrophic forgetting with respect to data from other clients and propose an efficient approach that modifies the cross-entropy objective on a per-client basis by re-weighting the softmax logits prior to computing the loss. This approach shields classes outside a client’s label set from abrupt representation change and we empirically demonstrate it can alleviate client forgetting and provide consistent improvements to standard federated learning algorithms. Our method is particularly beneficial under the most challenging federated learning settings where data heterogeneity is high and client participation in each round is low.

Cite this Paper

BibTeX
@InProceedings{pmlr-v232-legate23a, title = {Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated Learning}, author = {Legate, Gwen and Caccia, Lucas and Belilovsky, Eugene}, booktitle = {Proceedings of The 2nd Conference on Lifelong Learning Agents}, pages = {764--780}, year = {2023}, editor = {Chandar, Sarath and Pascanu, Razvan and Sedghi, Hanie and Precup, Doina}, volume = {232}, series = {Proceedings of Machine Learning Research}, month = {22--25 Aug}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v232/legate23a/legate23a.pdf}, url = {https://proceedings.mlr.press/v232/legate23a.html}, abstract = {In Federated Learning a global model is learned by aggregating model updates computed at a set of independent client nodes. To reduce communication costs, multiple gradient steps are performed at each node prior to aggregation. A key challenge in this setting is data heterogeneity across clients resulting in differing local objectives. This can lead clients to overly minimize their own local objective consequently diverging from the global solution. We demonstrate that individual client models experience a catastrophic forgetting with respect to data from other clients and propose an efficient approach that modifies the cross-entropy objective on a per-client basis by re-weighting the softmax logits prior to computing the loss. This approach shields classes outside a client’s label set from abrupt representation change and we empirically demonstrate it can alleviate client forgetting and provide consistent improvements to standard federated learning algorithms. Our method is particularly beneficial under the most challenging federated learning settings where data heterogeneity is high and client participation in each round is low.} }
Endnote
%0 Conference Paper %T Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated Learning %A Gwen Legate %A Lucas Caccia %A Eugene Belilovsky %B Proceedings of The 2nd Conference on Lifelong Learning Agents %C Proceedings of Machine Learning Research %D 2023 %E Sarath Chandar %E Razvan Pascanu %E Hanie Sedghi %E Doina Precup %F pmlr-v232-legate23a %I PMLR %P 764--780 %U https://proceedings.mlr.press/v232/legate23a.html %V 232 %X In Federated Learning a global model is learned by aggregating model updates computed at a set of independent client nodes. To reduce communication costs, multiple gradient steps are performed at each node prior to aggregation. A key challenge in this setting is data heterogeneity across clients resulting in differing local objectives. This can lead clients to overly minimize their own local objective consequently diverging from the global solution. We demonstrate that individual client models experience a catastrophic forgetting with respect to data from other clients and propose an efficient approach that modifies the cross-entropy objective on a per-client basis by re-weighting the softmax logits prior to computing the loss. This approach shields classes outside a client’s label set from abrupt representation change and we empirically demonstrate it can alleviate client forgetting and provide consistent improvements to standard federated learning algorithms. Our method is particularly beneficial under the most challenging federated learning settings where data heterogeneity is high and client participation in each round is low.
APA
Legate, G., Caccia, L. & Belilovsky, E.. (2023). Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated Learning. Proceedings of The 2nd Conference on Lifelong Learning Agents, in Proceedings of Machine Learning Research 232:764-780 Available from https://proceedings.mlr.press/v232/legate23a.html.

Related Material