Platonic Transformers: A Solid Choice For Equivariance

Mohammad Mohaiminul Islam, Rishabh Anand, David R Wessels, Friso De Kruiff, Thijs P. Kuipers, Zhitao Ying, Clara I. Sánchez, Sharvaree Vadgama, Georg Bökman, Erik J Bekkers
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:49939-49969, 2026.

Abstract

While widespread, Transformers lack inductive biases for geometric symmetries common in science and computer vision. Existing equivariant methods often sacrifice the efficiency and flexibility that make Transformers so effective through complex, computationally intensive designs. We introduce the Platonic Transformer to resolve this trade-off. By defining attention relative to reference frames from the Platonic solid symmetry groups, our method induces a principled weight-sharing scheme. This enables combined equivariance to continuous translations and Platonic symmetries, while preserving the exact architecture and computational cost of a standard Transformer. Furthermore, we show that this attention is formally equivalent to a dynamic group convolution, which reveals that the model learns adaptive geometric filters and enables a highly scalable, linear-time convolutional variant. Across diverse benchmarks in computer vision (CIFAR-10), 3D point clouds (ScanObjectNN), and molecular dynamics, property prediction and generation (OMol25, ProteinMD, QM9), the Platonic Transformer achieves competitive performance by leveraging these geometric constraints at no additional cost.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-islam26a, title = {Platonic Transformers: A Solid Choice For Equivariance}, author = {Islam, Mohammad Mohaiminul and Anand, Rishabh and Wessels, David R and De Kruiff, Friso and Kuipers, Thijs P. and Ying, Zhitao and S\'{a}nchez, Clara I. and Vadgama, Sharvaree and B\"{o}kman, Georg and Bekkers, Erik J}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {49939--49969}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/islam26a/islam26a.pdf}, url = {https://proceedings.mlr.press/v306/islam26a.html}, abstract = {While widespread, Transformers lack inductive biases for geometric symmetries common in science and computer vision. Existing equivariant methods often sacrifice the efficiency and flexibility that make Transformers so effective through complex, computationally intensive designs. We introduce the Platonic Transformer to resolve this trade-off. By defining attention relative to reference frames from the Platonic solid symmetry groups, our method induces a principled weight-sharing scheme. This enables combined equivariance to continuous translations and Platonic symmetries, while preserving the exact architecture and computational cost of a standard Transformer. Furthermore, we show that this attention is formally equivalent to a dynamic group convolution, which reveals that the model learns adaptive geometric filters and enables a highly scalable, linear-time convolutional variant. Across diverse benchmarks in computer vision (CIFAR-10), 3D point clouds (ScanObjectNN), and molecular dynamics, property prediction and generation (OMol25, ProteinMD, QM9), the Platonic Transformer achieves competitive performance by leveraging these geometric constraints at no additional cost.} }
Endnote
%0 Conference Paper %T Platonic Transformers: A Solid Choice For Equivariance %A Mohammad Mohaiminul Islam %A Rishabh Anand %A David R Wessels %A Friso De Kruiff %A Thijs P. Kuipers %A Zhitao Ying %A Clara I. Sánchez %A Sharvaree Vadgama %A Georg Bökman %A Erik J Bekkers %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-islam26a %I PMLR %P 49939--49969 %U https://proceedings.mlr.press/v306/islam26a.html %V 306 %X While widespread, Transformers lack inductive biases for geometric symmetries common in science and computer vision. Existing equivariant methods often sacrifice the efficiency and flexibility that make Transformers so effective through complex, computationally intensive designs. We introduce the Platonic Transformer to resolve this trade-off. By defining attention relative to reference frames from the Platonic solid symmetry groups, our method induces a principled weight-sharing scheme. This enables combined equivariance to continuous translations and Platonic symmetries, while preserving the exact architecture and computational cost of a standard Transformer. Furthermore, we show that this attention is formally equivalent to a dynamic group convolution, which reveals that the model learns adaptive geometric filters and enables a highly scalable, linear-time convolutional variant. Across diverse benchmarks in computer vision (CIFAR-10), 3D point clouds (ScanObjectNN), and molecular dynamics, property prediction and generation (OMol25, ProteinMD, QM9), the Platonic Transformer achieves competitive performance by leveraging these geometric constraints at no additional cost.
APA
Islam, M.M., Anand, R., Wessels, D.R., De Kruiff, F., Kuipers, T.P., Ying, Z., Sánchez, C.I., Vadgama, S., Bökman, G. & Bekkers, E.J.. (2026). Platonic Transformers: A Solid Choice For Equivariance. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:49939-49969 Available from https://proceedings.mlr.press/v306/islam26a.html.

Related Material