GNOT: A General Neural Operator Transformer for Operator Learning

Zhongkai Hao, Zhengyi Wang, Hang Su, Chengyang Ying, Yinpeng Dong, Songming Liu, Ze Cheng, Jian Song, Jun Zhu
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:12556-12569, 2023.

Abstract

Learning partial differential equations’ (PDEs) solution operators is an essential problem in machine learning. However, there are several challenges for learning operators in practical applications like the irregular mesh, multiple input functions, and complexity of the PDEs’ solution. To address these challenges, we propose a general neural operator transformer (GNOT), a scalable and effective transformer-based framework for learning operators. By designing a novel heterogeneous normalized attention layer, our model is highly flexible to handle multiple input functions and irregular meshes. Besides, we introduce a geometric gating mechanism which could be viewed as a soft domain decomposition to solve the multi-scale problems. The large model capacity of the transformer architecture grants our model the possibility to scale to large datasets and practical problems. We conduct extensive experiments on multiple challenging datasets from different domains and achieve a remarkable improvement compared with alternative methods. Our code and data are publicly available at https://github.com/thu-ml/GNOT.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-hao23c, title = {{GNOT}: A General Neural Operator Transformer for Operator Learning}, author = {Hao, Zhongkai and Wang, Zhengyi and Su, Hang and Ying, Chengyang and Dong, Yinpeng and Liu, Songming and Cheng, Ze and Song, Jian and Zhu, Jun}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {12556--12569}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/hao23c/hao23c.pdf}, url = {https://proceedings.mlr.press/v202/hao23c.html}, abstract = {Learning partial differential equations’ (PDEs) solution operators is an essential problem in machine learning. However, there are several challenges for learning operators in practical applications like the irregular mesh, multiple input functions, and complexity of the PDEs’ solution. To address these challenges, we propose a general neural operator transformer (GNOT), a scalable and effective transformer-based framework for learning operators. By designing a novel heterogeneous normalized attention layer, our model is highly flexible to handle multiple input functions and irregular meshes. Besides, we introduce a geometric gating mechanism which could be viewed as a soft domain decomposition to solve the multi-scale problems. The large model capacity of the transformer architecture grants our model the possibility to scale to large datasets and practical problems. We conduct extensive experiments on multiple challenging datasets from different domains and achieve a remarkable improvement compared with alternative methods. Our code and data are publicly available at https://github.com/thu-ml/GNOT.} }
Endnote
%0 Conference Paper %T GNOT: A General Neural Operator Transformer for Operator Learning %A Zhongkai Hao %A Zhengyi Wang %A Hang Su %A Chengyang Ying %A Yinpeng Dong %A Songming Liu %A Ze Cheng %A Jian Song %A Jun Zhu %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-hao23c %I PMLR %P 12556--12569 %U https://proceedings.mlr.press/v202/hao23c.html %V 202 %X Learning partial differential equations’ (PDEs) solution operators is an essential problem in machine learning. However, there are several challenges for learning operators in practical applications like the irregular mesh, multiple input functions, and complexity of the PDEs’ solution. To address these challenges, we propose a general neural operator transformer (GNOT), a scalable and effective transformer-based framework for learning operators. By designing a novel heterogeneous normalized attention layer, our model is highly flexible to handle multiple input functions and irregular meshes. Besides, we introduce a geometric gating mechanism which could be viewed as a soft domain decomposition to solve the multi-scale problems. The large model capacity of the transformer architecture grants our model the possibility to scale to large datasets and practical problems. We conduct extensive experiments on multiple challenging datasets from different domains and achieve a remarkable improvement compared with alternative methods. Our code and data are publicly available at https://github.com/thu-ml/GNOT.
APA
Hao, Z., Wang, Z., Su, H., Ying, C., Dong, Y., Liu, S., Cheng, Z., Song, J. & Zhu, J.. (2023). GNOT: A General Neural Operator Transformer for Operator Learning. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:12556-12569 Available from https://proceedings.mlr.press/v202/hao23c.html.

Related Material