PowerSoftmax: Towards Secure LLM Inference Over Encrypted Data

Itamar Zimerman, Allon Adir, Ehud Aharoni, Matan Avitan, Moran Baruch, Nir Drucker, Jenny Lerner, Ramy Masalha, Reut Moshe, Omri Soceanu
Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, PMLR 300:5203-5211, 2026.

Abstract

Modern cryptographic methods for implementing privacy-preserving LLMs such as HE require the LLMs to have a polynomial form. Forming such a representation is challenging because transformers include non-polynomial components, such as Softmax and layer normalization. Previous approaches have either directly approximated pre-trained models with large-degree polynomials, which are less efficient over HE, or replaced non-polynomial components with easier-to-approximate primitives before training, e.g., Softmax with pointwise attention. The latter approach might introduce scalability challenges. We present a new HE-friendly variant of self-attention that offers a stable form for training and is easy to approximate with polynomials for secure inference. Our work introduces the first polynomial LLMs over a billion parameters, exceeding the size of previous models by more than tenfold. The resulting models demonstrate reasoning and in-context learning (ICL) capabilities comparable to standard transformers of the same size, representing a breakthrough in the field. Finally, we provide a detailed latency breakdown for each computation over encrypted data, paving the way for further optimization, and explore the differences in inductive bias between models relying on our HE-friendly variant and standard transformers. Our code is attached as a supplement.

Cite this Paper

BibTeX
@InProceedings{pmlr-v300-zimerman26a, title = { PowerSoftmax: Towards Secure LLM Inference Over Encrypted Data }, author = {Zimerman, Itamar and Adir, Allon and Aharoni, Ehud and Avitan, Matan and Baruch, Moran and Drucker, Nir and Lerner, Jenny and Masalha, Ramy and Moshe, Reut and Soceanu, Omri}, booktitle = {Proceedings of The 29th International Conference on Artificial Intelligence and Statistics}, pages = {5203--5211}, year = {2026}, editor = {Khan, Emtiyaz and Li, Yingzhen and Solin, Arno and Ramdas, Aaditya}, volume = {300}, series = {Proceedings of Machine Learning Research}, month = {02--05 May}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v300/main/assets/zimerman26a/zimerman26a.pdf}, url = {https://proceedings.mlr.press/v300/zimerman26a.html}, abstract = { Modern cryptographic methods for implementing privacy-preserving LLMs such as HE require the LLMs to have a polynomial form. Forming such a representation is challenging because transformers include non-polynomial components, such as Softmax and layer normalization. Previous approaches have either directly approximated pre-trained models with large-degree polynomials, which are less efficient over HE, or replaced non-polynomial components with easier-to-approximate primitives before training, e.g., Softmax with pointwise attention. The latter approach might introduce scalability challenges. We present a new HE-friendly variant of self-attention that offers a stable form for training and is easy to approximate with polynomials for secure inference. Our work introduces the first polynomial LLMs over a billion parameters, exceeding the size of previous models by more than tenfold. The resulting models demonstrate reasoning and in-context learning (ICL) capabilities comparable to standard transformers of the same size, representing a breakthrough in the field. Finally, we provide a detailed latency breakdown for each computation over encrypted data, paving the way for further optimization, and explore the differences in inductive bias between models relying on our HE-friendly variant and standard transformers. Our code is attached as a supplement. } }
Endnote
%0 Conference Paper %T PowerSoftmax: Towards Secure LLM Inference Over Encrypted Data %A Itamar Zimerman %A Allon Adir %A Ehud Aharoni %A Matan Avitan %A Moran Baruch %A Nir Drucker %A Jenny Lerner %A Ramy Masalha %A Reut Moshe %A Omri Soceanu %B Proceedings of The 29th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2026 %E Emtiyaz Khan %E Yingzhen Li %E Arno Solin %E Aaditya Ramdas %F pmlr-v300-zimerman26a %I PMLR %P 5203--5211 %U https://proceedings.mlr.press/v300/zimerman26a.html %V 300 %X Modern cryptographic methods for implementing privacy-preserving LLMs such as HE require the LLMs to have a polynomial form. Forming such a representation is challenging because transformers include non-polynomial components, such as Softmax and layer normalization. Previous approaches have either directly approximated pre-trained models with large-degree polynomials, which are less efficient over HE, or replaced non-polynomial components with easier-to-approximate primitives before training, e.g., Softmax with pointwise attention. The latter approach might introduce scalability challenges. We present a new HE-friendly variant of self-attention that offers a stable form for training and is easy to approximate with polynomials for secure inference. Our work introduces the first polynomial LLMs over a billion parameters, exceeding the size of previous models by more than tenfold. The resulting models demonstrate reasoning and in-context learning (ICL) capabilities comparable to standard transformers of the same size, representing a breakthrough in the field. Finally, we provide a detailed latency breakdown for each computation over encrypted data, paving the way for further optimization, and explore the differences in inductive bias between models relying on our HE-friendly variant and standard transformers. Our code is attached as a supplement.
APA
Zimerman, I., Adir, A., Aharoni, E., Avitan, M., Baruch, M., Drucker, N., Lerner, J., Masalha, R., Moshe, R. & Soceanu, O.. (2026). PowerSoftmax: Towards Secure LLM Inference Over Encrypted Data . Proceedings of The 29th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 300:5203-5211 Available from https://proceedings.mlr.press/v300/zimerman26a.html.

Related Material