Walrus: A Cross-domain Foundation Model for Continuum Dynamics

Michael Mccabe, Payel Mukhopadhyay, Tanya Marwah, Bruno Régaldo-Saint Blancard, François Rozet, Cristiana Diaconu, Lucas Thibaut Meyer, Kaze W. K. Wong, Mohammad-Hadi Sotoudeh, Alberto Bietti, Irina Espejo Morales, Rio Alexa Fear, Siavash Golkar, Tom Hehir, Keiya Hirashima, Geraud Krawezik, Francois Lanusse, Rudy Morel, Ruben Ohana, Liam Holden Parker, Mariel Pettee, Jeff Shen, Kyunghyun Cho, Miles Cranmer, Shirley Ho
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:87204-87250, 2026.

Abstract

Foundation models have transformed machine learning for language and vision, but achieving comparable impact in physical simulation remains a challenge. Data heterogeneity and unstable long-term dynamics inhibit learning from sufficiently diverse dynamics, while varying resolutions and dimensionalities challenge efficient training on modern hardware. Through empirical and theoretical analysis, we incorporate new approaches to mitigate these obstacles, including a harmonic-analysis–based stabilization method, load-balanced distributed 2D-3D training strategies, and compute-adaptive tokenization. Using these tools, we develop Walrus, a transformer-based foundation model developed primarily for fluid-like continuum dynamics. Walrus is pretrained on nineteen diverse scenarios spanning astrophysics, geoscience, rheology, plasma physics, acoustics, and classical fluids. Experiments show that Walrus outperforms prior foundation models on both short- and long-term prediction horizons on downstream tasks and across the breadth of pretraining data, while ablation studies confirm the value of our contributions to forecast stability, training throughput, and transfer performance over conventional approaches.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-mccabe26a, title = {Walrus: A Cross-domain Foundation Model for Continuum Dynamics}, author = {Mccabe, Michael and Mukhopadhyay, Payel and Marwah, Tanya and R\'{e}galdo-Saint Blancard, Bruno and Rozet, Fran\c{c}ois and Diaconu, Cristiana and Meyer, Lucas Thibaut and Wong, Kaze W. K. and Sotoudeh, Mohammad-Hadi and Bietti, Alberto and Morales, Irina Espejo and Fear, Rio Alexa and Golkar, Siavash and Hehir, Tom and Hirashima, Keiya and Krawezik, Geraud and Lanusse, Francois and Morel, Rudy and Ohana, Ruben and Parker, Liam Holden and Pettee, Mariel and Shen, Jeff and Cho, Kyunghyun and Cranmer, Miles and Ho, Shirley}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {87204--87250}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/mccabe26a/mccabe26a.pdf}, url = {https://proceedings.mlr.press/v306/mccabe26a.html}, abstract = {Foundation models have transformed machine learning for language and vision, but achieving comparable impact in physical simulation remains a challenge. Data heterogeneity and unstable long-term dynamics inhibit learning from sufficiently diverse dynamics, while varying resolutions and dimensionalities challenge efficient training on modern hardware. Through empirical and theoretical analysis, we incorporate new approaches to mitigate these obstacles, including a harmonic-analysis–based stabilization method, load-balanced distributed 2D-3D training strategies, and compute-adaptive tokenization. Using these tools, we develop Walrus, a transformer-based foundation model developed primarily for fluid-like continuum dynamics. Walrus is pretrained on nineteen diverse scenarios spanning astrophysics, geoscience, rheology, plasma physics, acoustics, and classical fluids. Experiments show that Walrus outperforms prior foundation models on both short- and long-term prediction horizons on downstream tasks and across the breadth of pretraining data, while ablation studies confirm the value of our contributions to forecast stability, training throughput, and transfer performance over conventional approaches.} }
Endnote
%0 Conference Paper %T Walrus: A Cross-domain Foundation Model for Continuum Dynamics %A Michael Mccabe %A Payel Mukhopadhyay %A Tanya Marwah %A Bruno Régaldo-Saint Blancard %A François Rozet %A Cristiana Diaconu %A Lucas Thibaut Meyer %A Kaze W. K. Wong %A Mohammad-Hadi Sotoudeh %A Alberto Bietti %A Irina Espejo Morales %A Rio Alexa Fear %A Siavash Golkar %A Tom Hehir %A Keiya Hirashima %A Geraud Krawezik %A Francois Lanusse %A Rudy Morel %A Ruben Ohana %A Liam Holden Parker %A Mariel Pettee %A Jeff Shen %A Kyunghyun Cho %A Miles Cranmer %A Shirley Ho %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-mccabe26a %I PMLR %P 87204--87250 %U https://proceedings.mlr.press/v306/mccabe26a.html %V 306 %X Foundation models have transformed machine learning for language and vision, but achieving comparable impact in physical simulation remains a challenge. Data heterogeneity and unstable long-term dynamics inhibit learning from sufficiently diverse dynamics, while varying resolutions and dimensionalities challenge efficient training on modern hardware. Through empirical and theoretical analysis, we incorporate new approaches to mitigate these obstacles, including a harmonic-analysis–based stabilization method, load-balanced distributed 2D-3D training strategies, and compute-adaptive tokenization. Using these tools, we develop Walrus, a transformer-based foundation model developed primarily for fluid-like continuum dynamics. Walrus is pretrained on nineteen diverse scenarios spanning astrophysics, geoscience, rheology, plasma physics, acoustics, and classical fluids. Experiments show that Walrus outperforms prior foundation models on both short- and long-term prediction horizons on downstream tasks and across the breadth of pretraining data, while ablation studies confirm the value of our contributions to forecast stability, training throughput, and transfer performance over conventional approaches.
APA
Mccabe, M., Mukhopadhyay, P., Marwah, T., Régaldo-Saint Blancard, B., Rozet, F., Diaconu, C., Meyer, L.T., Wong, K.W.K., Sotoudeh, M., Bietti, A., Morales, I.E., Fear, R.A., Golkar, S., Hehir, T., Hirashima, K., Krawezik, G., Lanusse, F., Morel, R., Ohana, R., Parker, L.H., Pettee, M., Shen, J., Cho, K., Cranmer, M. & Ho, S.. (2026). Walrus: A Cross-domain Foundation Model for Continuum Dynamics. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:87204-87250 Available from https://proceedings.mlr.press/v306/mccabe26a.html.

Related Material