FedPAT: Federated Test-Time Adaptation via Prototype Affinity Topology

Shunxin Guo, Jiaqi Lv, Zhiqiang Kou, Shuxia Lin, Xin Geng
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:38109-38130, 2026.

Abstract

Federated Learning (FL) enables privacy-preserving collaboration among distributed clients in open-world environments, but its performance often degrades under data heterogeneity and unpredictable distribution shifts. Test-Time Adaptation (TTA) has recently been introduced into FL to leverage unlabeled data from unseen clients for online adaptation. However, most existing federated TTA methods employ local feature statistics, which can be brittle under diverse and severe distribution shifts. In this work, we observe that despite significant variations in feature distributions, the relational structure among class prototypes—termed prototype affinity topology—remains remarkably stable across heterogeneous clients. Building on this insight, we propose FedPAT, a Federated TTA framework that leverages Prototype Affinity Topology as a cross-client structural prior. FedPAT learns a global PAT by aggregating class prototypes from source clients, capturing consensus inter-class relationships that are robust to local distribution variations. For unseen target clients, we design a topology-aware mechanism that enhances predictions via diffusion of the global PAT, fuses them with parametric outputs, and performs lightweight optimization for robust test-time adaptation. Extensive experiments demonstrate that FedPAT consistently outperforms advanced federated TTA and classical TTA methods across various distribution shifts.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-guo26m, title = {{F}ed{PAT}: Federated Test-Time Adaptation via Prototype Affinity Topology}, author = {Guo, Shunxin and Lv, Jiaqi and Kou, Zhiqiang and Lin, Shuxia and Geng, Xin}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {38109--38130}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/guo26m/guo26m.pdf}, url = {https://proceedings.mlr.press/v306/guo26m.html}, abstract = {Federated Learning (FL) enables privacy-preserving collaboration among distributed clients in open-world environments, but its performance often degrades under data heterogeneity and unpredictable distribution shifts. Test-Time Adaptation (TTA) has recently been introduced into FL to leverage unlabeled data from unseen clients for online adaptation. However, most existing federated TTA methods employ local feature statistics, which can be brittle under diverse and severe distribution shifts. In this work, we observe that despite significant variations in feature distributions, the relational structure among class prototypes—termed prototype affinity topology—remains remarkably stable across heterogeneous clients. Building on this insight, we propose FedPAT, a Federated TTA framework that leverages Prototype Affinity Topology as a cross-client structural prior. FedPAT learns a global PAT by aggregating class prototypes from source clients, capturing consensus inter-class relationships that are robust to local distribution variations. For unseen target clients, we design a topology-aware mechanism that enhances predictions via diffusion of the global PAT, fuses them with parametric outputs, and performs lightweight optimization for robust test-time adaptation. Extensive experiments demonstrate that FedPAT consistently outperforms advanced federated TTA and classical TTA methods across various distribution shifts.} }
Endnote
%0 Conference Paper %T FedPAT: Federated Test-Time Adaptation via Prototype Affinity Topology %A Shunxin Guo %A Jiaqi Lv %A Zhiqiang Kou %A Shuxia Lin %A Xin Geng %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-guo26m %I PMLR %P 38109--38130 %U https://proceedings.mlr.press/v306/guo26m.html %V 306 %X Federated Learning (FL) enables privacy-preserving collaboration among distributed clients in open-world environments, but its performance often degrades under data heterogeneity and unpredictable distribution shifts. Test-Time Adaptation (TTA) has recently been introduced into FL to leverage unlabeled data from unseen clients for online adaptation. However, most existing federated TTA methods employ local feature statistics, which can be brittle under diverse and severe distribution shifts. In this work, we observe that despite significant variations in feature distributions, the relational structure among class prototypes—termed prototype affinity topology—remains remarkably stable across heterogeneous clients. Building on this insight, we propose FedPAT, a Federated TTA framework that leverages Prototype Affinity Topology as a cross-client structural prior. FedPAT learns a global PAT by aggregating class prototypes from source clients, capturing consensus inter-class relationships that are robust to local distribution variations. For unseen target clients, we design a topology-aware mechanism that enhances predictions via diffusion of the global PAT, fuses them with parametric outputs, and performs lightweight optimization for robust test-time adaptation. Extensive experiments demonstrate that FedPAT consistently outperforms advanced federated TTA and classical TTA methods across various distribution shifts.
APA
Guo, S., Lv, J., Kou, Z., Lin, S. & Geng, X.. (2026). FedPAT: Federated Test-Time Adaptation via Prototype Affinity Topology. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:38109-38130 Available from https://proceedings.mlr.press/v306/guo26m.html.

Related Material