EMFormer: Efficient Multi-Scale Transformer for Accumulative Context Weather Forecasting

Hao Chen, Tao Han, Jie Zhang, Song Guo, Fenghua Ling, Lei Bai
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:14464-14507, 2026.

Abstract

Long-term weather forecasting is critical for socioeconomic planning and disaster preparedness. While recent approaches employ finetuning to extend prediction horizons, they remain constrained by the issues of catastrophic forgetting, error accumulation, and high training overhead. To address these limitations, we present a novel pipeline across pretraining, finetuning and forecasting to enhance long-context modeling while reducing computational overhead. First, we introduce an Efficient Multi-scale Transformer (EMFormer) to extract multi-scale features through a single convolution in both training and inference. Based on the new architecture, we further employ an accumulative context finetuning to improve temporal consistency without degrading short-term accuracy. Additionally, we propose a composite loss that dynamically balances different terms via a sinusoidal weighting, thereby adaptively guiding the optimization trajectory throughout pretraining and finetuning. Experiments show that our approach achieves great performance in weather forecasting and extreme event prediction, substantially improving long-term forecast accuracy. Moreover, EMFormer demonstrates strong generalization on vision benchmarks (ImageNet-1K and ADE20K). Code: https://github.com/chenhao-zju/emformer

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-chen26ao, title = {{EMF}ormer: Efficient Multi-Scale Transformer for Accumulative Context Weather Forecasting}, author = {Chen, Hao and Han, Tao and Zhang, Jie and Guo, Song and Ling, Fenghua and Bai, Lei}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {14464--14507}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/chen26ao/chen26ao.pdf}, url = {https://proceedings.mlr.press/v306/chen26ao.html}, abstract = {Long-term weather forecasting is critical for socioeconomic planning and disaster preparedness. While recent approaches employ finetuning to extend prediction horizons, they remain constrained by the issues of catastrophic forgetting, error accumulation, and high training overhead. To address these limitations, we present a novel pipeline across pretraining, finetuning and forecasting to enhance long-context modeling while reducing computational overhead. First, we introduce an Efficient Multi-scale Transformer (EMFormer) to extract multi-scale features through a single convolution in both training and inference. Based on the new architecture, we further employ an accumulative context finetuning to improve temporal consistency without degrading short-term accuracy. Additionally, we propose a composite loss that dynamically balances different terms via a sinusoidal weighting, thereby adaptively guiding the optimization trajectory throughout pretraining and finetuning. Experiments show that our approach achieves great performance in weather forecasting and extreme event prediction, substantially improving long-term forecast accuracy. Moreover, EMFormer demonstrates strong generalization on vision benchmarks (ImageNet-1K and ADE20K). Code: https://github.com/chenhao-zju/emformer} }
Endnote
%0 Conference Paper %T EMFormer: Efficient Multi-Scale Transformer for Accumulative Context Weather Forecasting %A Hao Chen %A Tao Han %A Jie Zhang %A Song Guo %A Fenghua Ling %A Lei Bai %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-chen26ao %I PMLR %P 14464--14507 %U https://proceedings.mlr.press/v306/chen26ao.html %V 306 %X Long-term weather forecasting is critical for socioeconomic planning and disaster preparedness. While recent approaches employ finetuning to extend prediction horizons, they remain constrained by the issues of catastrophic forgetting, error accumulation, and high training overhead. To address these limitations, we present a novel pipeline across pretraining, finetuning and forecasting to enhance long-context modeling while reducing computational overhead. First, we introduce an Efficient Multi-scale Transformer (EMFormer) to extract multi-scale features through a single convolution in both training and inference. Based on the new architecture, we further employ an accumulative context finetuning to improve temporal consistency without degrading short-term accuracy. Additionally, we propose a composite loss that dynamically balances different terms via a sinusoidal weighting, thereby adaptively guiding the optimization trajectory throughout pretraining and finetuning. Experiments show that our approach achieves great performance in weather forecasting and extreme event prediction, substantially improving long-term forecast accuracy. Moreover, EMFormer demonstrates strong generalization on vision benchmarks (ImageNet-1K and ADE20K). Code: https://github.com/chenhao-zju/emformer
APA
Chen, H., Han, T., Zhang, J., Guo, S., Ling, F. & Bai, L.. (2026). EMFormer: Efficient Multi-Scale Transformer for Accumulative Context Weather Forecasting. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:14464-14507 Available from https://proceedings.mlr.press/v306/chen26ao.html.

Related Material