The Shape of Addition: Geometric Structures of Arithmetic in Large Language Models

Liuyuan Wen, Xun Zhu, Lihao Huang, Wenbin Li, Yang Gao
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:134748-134776, 2026.

Abstract

Large Language Models exhibit paradoxical fragility in fundamental arithmetic, implying a disconnect between internal computation and discrete output. By analyzing the residual stream geometry during multi-operand addition, we identify the Iso-Raw-Sum Trajectory (IRST), a topological manifold where representations are anchored by semantic digits and modulated by continuous carry fibers. We propose the Noisy Quantization Model, which frames arithmetic errors as topological slippages caused by internal neural noise pushing a continuous, latent carry potential across quantization thresholds. This geometric framework further elucidates probe versatility, explaining how lightweight probes can disentangle conflicting latent signals (such as ground truth versus hallucination) from a single activation vector. Finally, we validate these insights through a geometric consistency check method that effectively detects and corrects these quantization failures during inference.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-wen26l, title = {The Shape of Addition: Geometric Structures of Arithmetic in Large Language Models}, author = {Wen, Liuyuan and Zhu, Xun and Huang, Lihao and Li, Wenbin and Gao, Yang}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {134748--134776}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/wen26l/wen26l.pdf}, url = {https://proceedings.mlr.press/v306/wen26l.html}, abstract = {Large Language Models exhibit paradoxical fragility in fundamental arithmetic, implying a disconnect between internal computation and discrete output. By analyzing the residual stream geometry during multi-operand addition, we identify the Iso-Raw-Sum Trajectory (IRST), a topological manifold where representations are anchored by semantic digits and modulated by continuous carry fibers. We propose the Noisy Quantization Model, which frames arithmetic errors as topological slippages caused by internal neural noise pushing a continuous, latent carry potential across quantization thresholds. This geometric framework further elucidates probe versatility, explaining how lightweight probes can disentangle conflicting latent signals (such as ground truth versus hallucination) from a single activation vector. Finally, we validate these insights through a geometric consistency check method that effectively detects and corrects these quantization failures during inference.} }
Endnote
%0 Conference Paper %T The Shape of Addition: Geometric Structures of Arithmetic in Large Language Models %A Liuyuan Wen %A Xun Zhu %A Lihao Huang %A Wenbin Li %A Yang Gao %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-wen26l %I PMLR %P 134748--134776 %U https://proceedings.mlr.press/v306/wen26l.html %V 306 %X Large Language Models exhibit paradoxical fragility in fundamental arithmetic, implying a disconnect between internal computation and discrete output. By analyzing the residual stream geometry during multi-operand addition, we identify the Iso-Raw-Sum Trajectory (IRST), a topological manifold where representations are anchored by semantic digits and modulated by continuous carry fibers. We propose the Noisy Quantization Model, which frames arithmetic errors as topological slippages caused by internal neural noise pushing a continuous, latent carry potential across quantization thresholds. This geometric framework further elucidates probe versatility, explaining how lightweight probes can disentangle conflicting latent signals (such as ground truth versus hallucination) from a single activation vector. Finally, we validate these insights through a geometric consistency check method that effectively detects and corrects these quantization failures during inference.
APA
Wen, L., Zhu, X., Huang, L., Li, W. & Gao, Y.. (2026). The Shape of Addition: Geometric Structures of Arithmetic in Large Language Models. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:134748-134776 Available from https://proceedings.mlr.press/v306/wen26l.html.

Related Material