Learning High-Frequency Functions Made Easy with Sinusoidal Positional Encoding

Chuanhao Sun, Zhihang Yuan, Kai Xu, Luo Mai, Siddharth N, Shuo Chen, Mahesh K. Marina
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:47218-47233, 2024.

Abstract

Fourier features based positional encoding (PE) is commonly used in machine learning tasks that involve learning high-frequency features from low-dimensional inputs, such as 3D view synthesis and time series regression with neural tangent kernels. Despite their effectiveness, existing PEs require manual, empirical adjustment of crucial hyperparameters, specifically the Fourier features, tailored to each unique task. Further, PEs face challenges in efficiently learning high-frequency functions, particularly in tasks with limited data. In this paper, we introduce sinusoidal PE (SPE), designed to efficiently learn adaptive frequency features closely aligned with the true underlying function. Our experiments demonstrate that SPE, without hyperparameter tuning, consistently achieves enhanced fidelity and faster training across various tasks, including 3D view synthesis, Text-to-Speech generation, and 1D regression. SPE is implemented as a direct replacement for existing PEs. Its plug-and-play nature lets numerous tasks easily adopt and benefit from SPE.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-sun24m, title = {Learning High-Frequency Functions Made Easy with Sinusoidal Positional Encoding}, author = {Sun, Chuanhao and Yuan, Zhihang and Xu, Kai and Mai, Luo and N, Siddharth and Chen, Shuo and Marina, Mahesh K.}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {47218--47233}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/sun24m/sun24m.pdf}, url = {https://proceedings.mlr.press/v235/sun24m.html}, abstract = {Fourier features based positional encoding (PE) is commonly used in machine learning tasks that involve learning high-frequency features from low-dimensional inputs, such as 3D view synthesis and time series regression with neural tangent kernels. Despite their effectiveness, existing PEs require manual, empirical adjustment of crucial hyperparameters, specifically the Fourier features, tailored to each unique task. Further, PEs face challenges in efficiently learning high-frequency functions, particularly in tasks with limited data. In this paper, we introduce sinusoidal PE (SPE), designed to efficiently learn adaptive frequency features closely aligned with the true underlying function. Our experiments demonstrate that SPE, without hyperparameter tuning, consistently achieves enhanced fidelity and faster training across various tasks, including 3D view synthesis, Text-to-Speech generation, and 1D regression. SPE is implemented as a direct replacement for existing PEs. Its plug-and-play nature lets numerous tasks easily adopt and benefit from SPE.} }
Endnote
%0 Conference Paper %T Learning High-Frequency Functions Made Easy with Sinusoidal Positional Encoding %A Chuanhao Sun %A Zhihang Yuan %A Kai Xu %A Luo Mai %A Siddharth N %A Shuo Chen %A Mahesh K. Marina %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-sun24m %I PMLR %P 47218--47233 %U https://proceedings.mlr.press/v235/sun24m.html %V 235 %X Fourier features based positional encoding (PE) is commonly used in machine learning tasks that involve learning high-frequency features from low-dimensional inputs, such as 3D view synthesis and time series regression with neural tangent kernels. Despite their effectiveness, existing PEs require manual, empirical adjustment of crucial hyperparameters, specifically the Fourier features, tailored to each unique task. Further, PEs face challenges in efficiently learning high-frequency functions, particularly in tasks with limited data. In this paper, we introduce sinusoidal PE (SPE), designed to efficiently learn adaptive frequency features closely aligned with the true underlying function. Our experiments demonstrate that SPE, without hyperparameter tuning, consistently achieves enhanced fidelity and faster training across various tasks, including 3D view synthesis, Text-to-Speech generation, and 1D regression. SPE is implemented as a direct replacement for existing PEs. Its plug-and-play nature lets numerous tasks easily adopt and benefit from SPE.
APA
Sun, C., Yuan, Z., Xu, K., Mai, L., N, S., Chen, S. & Marina, M.K.. (2024). Learning High-Frequency Functions Made Easy with Sinusoidal Positional Encoding. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:47218-47233 Available from https://proceedings.mlr.press/v235/sun24m.html.

Related Material