SpikingLM: Towards Fully Spiking Language Model

Yu Liang, Zijian Zhou, Wenjie Wei, Shuai Wang, Honglin Cao, Ammar Belatreche, Yu Yang, Malu Zhang, Yang Yang, Haizhou Li
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:73744-73762, 2026.

Abstract

Leveraging event-driven computation mechanism, Spiking Neural Networks (SNNs) have emerged as a representative paradigm for energy-efficient edge intelligence. However, extending SNNs to modern deep language models still faces two fundamental challenges. First, dead neurons in deep SNNs lead to degraded gradients, limiting the training effectiveness of spiking language models. Second, removing Softmax for energy efficiency weakens token-wise competition, reducing the model’s ability to select salient tokens. To address these challenges, we propose Spiking Language Model (SpikingLM) to bridge the efficiency of SNNs and the capability of modern language models through two key innovations. First, we propose Distribution-aware Scaling method, which rescales linear outputs into an activation-friendly range to alleviate dead neurons and stabilize gradient propagation. Notably, its scaling parameters can be fused into the preceding linear layers, incurring no additional inference overhead. Second, we introduce Spike2Max to restore winner-takes-all mechanism via base-2 exponentiation and max-subtraction. Compared with Softmax, Spike2Max reduces energy consumption by over 95% using hardware-efficient bit-shift operations. Experiments show that SpikingLM reduces energy consumption by 57.9% and achieves state-of-the-art performance on GLUE, laying a promising foundation for energy-efficient language modeling. Code is available at https://github.com/hamings1/SpikingLM.git.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-liang26ae, title = {{S}piking{LM}: Towards Fully Spiking Language Model}, author = {Liang, Yu and Zhou, Zijian and Wei, Wenjie and Wang, Shuai and Cao, Honglin and Belatreche, Ammar and Yang, Yu and Zhang, Malu and Yang, Yang and Li, Haizhou}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {73744--73762}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/liang26ae/liang26ae.pdf}, url = {https://proceedings.mlr.press/v306/liang26ae.html}, abstract = {Leveraging event-driven computation mechanism, Spiking Neural Networks (SNNs) have emerged as a representative paradigm for energy-efficient edge intelligence. However, extending SNNs to modern deep language models still faces two fundamental challenges. First, dead neurons in deep SNNs lead to degraded gradients, limiting the training effectiveness of spiking language models. Second, removing Softmax for energy efficiency weakens token-wise competition, reducing the model’s ability to select salient tokens. To address these challenges, we propose Spiking Language Model (SpikingLM) to bridge the efficiency of SNNs and the capability of modern language models through two key innovations. First, we propose Distribution-aware Scaling method, which rescales linear outputs into an activation-friendly range to alleviate dead neurons and stabilize gradient propagation. Notably, its scaling parameters can be fused into the preceding linear layers, incurring no additional inference overhead. Second, we introduce Spike2Max to restore winner-takes-all mechanism via base-2 exponentiation and max-subtraction. Compared with Softmax, Spike2Max reduces energy consumption by over 95% using hardware-efficient bit-shift operations. Experiments show that SpikingLM reduces energy consumption by 57.9% and achieves state-of-the-art performance on GLUE, laying a promising foundation for energy-efficient language modeling. Code is available at https://github.com/hamings1/SpikingLM.git.} }
Endnote
%0 Conference Paper %T SpikingLM: Towards Fully Spiking Language Model %A Yu Liang %A Zijian Zhou %A Wenjie Wei %A Shuai Wang %A Honglin Cao %A Ammar Belatreche %A Yu Yang %A Malu Zhang %A Yang Yang %A Haizhou Li %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-liang26ae %I PMLR %P 73744--73762 %U https://proceedings.mlr.press/v306/liang26ae.html %V 306 %X Leveraging event-driven computation mechanism, Spiking Neural Networks (SNNs) have emerged as a representative paradigm for energy-efficient edge intelligence. However, extending SNNs to modern deep language models still faces two fundamental challenges. First, dead neurons in deep SNNs lead to degraded gradients, limiting the training effectiveness of spiking language models. Second, removing Softmax for energy efficiency weakens token-wise competition, reducing the model’s ability to select salient tokens. To address these challenges, we propose Spiking Language Model (SpikingLM) to bridge the efficiency of SNNs and the capability of modern language models through two key innovations. First, we propose Distribution-aware Scaling method, which rescales linear outputs into an activation-friendly range to alleviate dead neurons and stabilize gradient propagation. Notably, its scaling parameters can be fused into the preceding linear layers, incurring no additional inference overhead. Second, we introduce Spike2Max to restore winner-takes-all mechanism via base-2 exponentiation and max-subtraction. Compared with Softmax, Spike2Max reduces energy consumption by over 95% using hardware-efficient bit-shift operations. Experiments show that SpikingLM reduces energy consumption by 57.9% and achieves state-of-the-art performance on GLUE, laying a promising foundation for energy-efficient language modeling. Code is available at https://github.com/hamings1/SpikingLM.git.
APA
Liang, Y., Zhou, Z., Wei, W., Wang, S., Cao, H., Belatreche, A., Yang, Y., Zhang, M., Yang, Y. & Li, H.. (2026). SpikingLM: Towards Fully Spiking Language Model. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:73744-73762 Available from https://proceedings.mlr.press/v306/liang26ae.html.

Related Material