Impact of Positional Encoding: Clean and Adversarial Rademacher Complexity for Transformers under In-Context Regression

Weiyi He, Yue Xing
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:4438-4446, 2026.

Abstract

Positional encoding (PE) is a core architectural component of Transformers, yet its impact on the Transformer’s generalization and robustness remains unclear. In this work, we provide the first generalization analysis for single-layer Transformer under in-context regression that explicitly accounts for a trainable PE module. Our result shows that PE systematically enlarges the generalization gap. Extending to the adversarial setting, we derive the adversarial Rademacher generalization bound. We find that the gap between models with and without PE is magnified under attack, demonstrating that PE amplifies the vulnerability of models. Our bounds are empirically validated by a simulation study. Together, this work establishes a new framework for understanding the clean and adversarial generalization in ICL with PE.

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-he26d, title = { Impact of Positional Encoding: Clean and Adversarial Rademacher Complexity for Transformers under In-Context Regression }, author = {He, Weiyi and Xing, Yue}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {4438--4446}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/he26d/he26d.pdf}, url = {https://proceedings.mlr.press/v300/he26d.html}, abstract = { Positional encoding (PE) is a core architectural component of Transformers, yet its impact on the Transformer’s generalization and robustness remains unclear. In this work, we provide the first generalization analysis for single-layer Transformer under in-context regression that explicitly accounts for a trainable PE module. Our result shows that PE systematically enlarges the generalization gap. Extending to the adversarial setting, we derive the adversarial Rademacher generalization bound. We find that the gap between models with and without PE is magnified under attack, demonstrating that PE amplifies the vulnerability of models. Our bounds are empirically validated by a simulation study. Together, this work establishes a new framework for understanding the clean and adversarial generalization in ICL with PE. } }
Endnote
%0 Conference Paper %T Impact of Positional Encoding: Clean and Adversarial Rademacher Complexity for Transformers under In-Context Regression %A Weiyi He %A Yue Xing %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-he26d %I PMLR %P 4438--4446 %U https://proceedings.mlr.press/v300/he26d.html %V 300 %X Positional encoding (PE) is a core architectural component of Transformers, yet its impact on the Transformer’s generalization and robustness remains unclear. In this work, we provide the first generalization analysis for single-layer Transformer under in-context regression that explicitly accounts for a trainable PE module. Our result shows that PE systematically enlarges the generalization gap. Extending to the adversarial setting, we derive the adversarial Rademacher generalization bound. We find that the gap between models with and without PE is magnified under attack, demonstrating that PE amplifies the vulnerability of models. Our bounds are empirically validated by a simulation study. Together, this work establishes a new framework for understanding the clean and adversarial generalization in ICL with PE.
APA
He, W. & Xing, Y.. (2026). Impact of Positional Encoding: Clean and Adversarial Rademacher Complexity for Transformers under In-Context Regression . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:4438-4446 Available from https://proceedings.mlr.press/v300/he26d.html.

Related Material