Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation

Zhenyu He, Guhao Feng, Shengjie Luo, Kai Yang, Liwei Wang, Jingjing Xu, Zhi Zhang, Hongxia Yang, Di He
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:17858-17876, 2024.

Abstract

In this work, we leverage the intrinsic segmentation of language sequences and design a new positional encoding method called Bilevel Positional Encoding (BiPE). For each position, our BiPE blends an intra-segment encoding and an inter-segment encoding. The intra-segment encoding identifies the locations within a segment and helps the model capture the semantic information therein via absolute positional encoding. The inter-segment encoding specifies the segment index, models the relationships between segments, and aims to improve extrapolation capabilities via relative positional encoding. Theoretical analysis shows this disentanglement of positional information makes learning more effective. The empirical results also show that our BiPE has superior length extrapolation capabilities across a wide range of tasks in diverse text modalities.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-he24c, title = {Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation}, author = {He, Zhenyu and Feng, Guhao and Luo, Shengjie and Yang, Kai and Wang, Liwei and Xu, Jingjing and Zhang, Zhi and Yang, Hongxia and He, Di}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {17858--17876}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/he24c/he24c.pdf}, url = {https://proceedings.mlr.press/v235/he24c.html}, abstract = {In this work, we leverage the intrinsic segmentation of language sequences and design a new positional encoding method called Bilevel Positional Encoding (BiPE). For each position, our BiPE blends an intra-segment encoding and an inter-segment encoding. The intra-segment encoding identifies the locations within a segment and helps the model capture the semantic information therein via absolute positional encoding. The inter-segment encoding specifies the segment index, models the relationships between segments, and aims to improve extrapolation capabilities via relative positional encoding. Theoretical analysis shows this disentanglement of positional information makes learning more effective. The empirical results also show that our BiPE has superior length extrapolation capabilities across a wide range of tasks in diverse text modalities.} }
Endnote
%0 Conference Paper %T Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation %A Zhenyu He %A Guhao Feng %A Shengjie Luo %A Kai Yang %A Liwei Wang %A Jingjing Xu %A Zhi Zhang %A Hongxia Yang %A Di He %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-he24c %I PMLR %P 17858--17876 %U https://proceedings.mlr.press/v235/he24c.html %V 235 %X In this work, we leverage the intrinsic segmentation of language sequences and design a new positional encoding method called Bilevel Positional Encoding (BiPE). For each position, our BiPE blends an intra-segment encoding and an inter-segment encoding. The intra-segment encoding identifies the locations within a segment and helps the model capture the semantic information therein via absolute positional encoding. The inter-segment encoding specifies the segment index, models the relationships between segments, and aims to improve extrapolation capabilities via relative positional encoding. Theoretical analysis shows this disentanglement of positional information makes learning more effective. The empirical results also show that our BiPE has superior length extrapolation capabilities across a wide range of tasks in diverse text modalities.
APA
He, Z., Feng, G., Luo, S., Yang, K., Wang, L., Xu, J., Zhang, Z., Yang, H. & He, D.. (2024). Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:17858-17876 Available from https://proceedings.mlr.press/v235/he24c.html.

Related Material