Decoupling The "What" and "Where" With Polar Coordinate Positional Embedding

Anand Gopalakrishnan, Róbert Csordás, Jürgen Schmidhuber, Michael Curtis Mozer
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:36197-36214, 2026.

Abstract

The attention mechanism in a Transformer architecture matches key to query based on both content—the what—and position in a sequence—the where. We present an analysis indicating that what and where are entangled in the popular rotary position embedding (RoPE). This entanglement can impair performance particularly when decisions require independent matches on these two factors. We propose an improvement to RoPE, which we call Polar Coordinate Position Embedding or PoPE, that eliminates the what-where confound. PoPE is far superior on a diagnostic task requiring indexing solely by position or by content. On autoregressive sequence modeling in music, genomic, and natural language domains, Transformers using PoPE as the positional encoding scheme outperform baselines using RoPE with respect to evaluation loss (perplexity) and downstream task performance. On language modeling, these gains persist across model scale, from 124M to 774M parameters. Crucially, PoPE shows strong zero-shot length extrapolation capabilities compared not only to RoPE but even a method designed for extrapolation, YaRN, which requires additional fine tuning and frequency interpolation.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-gopalakrishnan26a, title = {Decoupling The "{W}hat" and "{W}here" With Polar Coordinate Positional Embedding}, author = {Gopalakrishnan, Anand and Csord\'{a}s, R\'{o}bert and Schmidhuber, J\"{u}rgen and Mozer, Michael Curtis}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {36197--36214}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/gopalakrishnan26a/gopalakrishnan26a.pdf}, url = {https://proceedings.mlr.press/v306/gopalakrishnan26a.html}, abstract = {The attention mechanism in a Transformer architecture matches key to query based on both content—the what—and position in a sequence—the where. We present an analysis indicating that what and where are entangled in the popular rotary position embedding (RoPE). This entanglement can impair performance particularly when decisions require independent matches on these two factors. We propose an improvement to RoPE, which we call Polar Coordinate Position Embedding or PoPE, that eliminates the what-where confound. PoPE is far superior on a diagnostic task requiring indexing solely by position or by content. On autoregressive sequence modeling in music, genomic, and natural language domains, Transformers using PoPE as the positional encoding scheme outperform baselines using RoPE with respect to evaluation loss (perplexity) and downstream task performance. On language modeling, these gains persist across model scale, from 124M to 774M parameters. Crucially, PoPE shows strong zero-shot length extrapolation capabilities compared not only to RoPE but even a method designed for extrapolation, YaRN, which requires additional fine tuning and frequency interpolation.} }
Endnote
%0 Conference Paper %T Decoupling The "What" and "Where" With Polar Coordinate Positional Embedding %A Anand Gopalakrishnan %A Róbert Csordás %A Jürgen Schmidhuber %A Michael Curtis Mozer %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-gopalakrishnan26a %I PMLR %P 36197--36214 %U https://proceedings.mlr.press/v306/gopalakrishnan26a.html %V 306 %X The attention mechanism in a Transformer architecture matches key to query based on both content—the what—and position in a sequence—the where. We present an analysis indicating that what and where are entangled in the popular rotary position embedding (RoPE). This entanglement can impair performance particularly when decisions require independent matches on these two factors. We propose an improvement to RoPE, which we call Polar Coordinate Position Embedding or PoPE, that eliminates the what-where confound. PoPE is far superior on a diagnostic task requiring indexing solely by position or by content. On autoregressive sequence modeling in music, genomic, and natural language domains, Transformers using PoPE as the positional encoding scheme outperform baselines using RoPE with respect to evaluation loss (perplexity) and downstream task performance. On language modeling, these gains persist across model scale, from 124M to 774M parameters. Crucially, PoPE shows strong zero-shot length extrapolation capabilities compared not only to RoPE but even a method designed for extrapolation, YaRN, which requires additional fine tuning and frequency interpolation.
APA
Gopalakrishnan, A., Csordás, R., Schmidhuber, J. & Mozer, M.C.. (2026). Decoupling The "What" and "Where" With Polar Coordinate Positional Embedding. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:36197-36214 Available from https://proceedings.mlr.press/v306/gopalakrishnan26a.html.

Related Material