mmBERT: A Modern Multilingual Encoder with Annealed Language Learning

Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, Benjamin Van Durme
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:86768-86783, 2026.

Abstract

Encoder-only language models are frequently used for a variety of language tasks, including classification and retrieval. However, there has been a lack of recent research efforts for encoder models, especially with respect to multilingual models. We introduce mmBERT, an encoder-only language model pretrained on 3T tokens of multilingual text in over 1800 languages. To build mmBERT we introduce several novel elements to massively multilingual encoder training, including phased data curation and scheduled language inclusion. We add over 1700 low-resource languages to the data mix only during the decay phase, showing that it boosts performance dramatically and maximizes the gains from the relatively small amount of training data without excessive repetition. The model uses recent advances in architecture and training schemes to be faster and more multilingual than other models and we release weights, data, and code. We show that mmBERT significantly outperforms the previous generation, on various tasks, for both high and low-resource languages.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-marone26a, title = {mm{BERT}: A Modern Multilingual Encoder with Annealed Language Learning}, author = {Marone, Marc and Weller, Orion and Fleshman, William and Yang, Eugene and Lawrie, Dawn and Van Durme, Benjamin}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {86768--86783}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/marone26a/marone26a.pdf}, url = {https://proceedings.mlr.press/v306/marone26a.html}, abstract = {Encoder-only language models are frequently used for a variety of language tasks, including classification and retrieval. However, there has been a lack of recent research efforts for encoder models, especially with respect to multilingual models. We introduce mmBERT, an encoder-only language model pretrained on 3T tokens of multilingual text in over 1800 languages. To build mmBERT we introduce several novel elements to massively multilingual encoder training, including phased data curation and scheduled language inclusion. We add over 1700 low-resource languages to the data mix only during the decay phase, showing that it boosts performance dramatically and maximizes the gains from the relatively small amount of training data without excessive repetition. The model uses recent advances in architecture and training schemes to be faster and more multilingual than other models and we release weights, data, and code. We show that mmBERT significantly outperforms the previous generation, on various tasks, for both high and low-resource languages.} }
Endnote
%0 Conference Paper %T mmBERT: A Modern Multilingual Encoder with Annealed Language Learning %A Marc Marone %A Orion Weller %A William Fleshman %A Eugene Yang %A Dawn Lawrie %A Benjamin Van Durme %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-marone26a %I PMLR %P 86768--86783 %U https://proceedings.mlr.press/v306/marone26a.html %V 306 %X Encoder-only language models are frequently used for a variety of language tasks, including classification and retrieval. However, there has been a lack of recent research efforts for encoder models, especially with respect to multilingual models. We introduce mmBERT, an encoder-only language model pretrained on 3T tokens of multilingual text in over 1800 languages. To build mmBERT we introduce several novel elements to massively multilingual encoder training, including phased data curation and scheduled language inclusion. We add over 1700 low-resource languages to the data mix only during the decay phase, showing that it boosts performance dramatically and maximizes the gains from the relatively small amount of training data without excessive repetition. The model uses recent advances in architecture and training schemes to be faster and more multilingual than other models and we release weights, data, and code. We show that mmBERT significantly outperforms the previous generation, on various tasks, for both high and low-resource languages.
APA
Marone, M., Weller, O., Fleshman, W., Yang, E., Lawrie, D. & Van Durme, B.. (2026). mmBERT: A Modern Multilingual Encoder with Annealed Language Learning. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:86768-86783 Available from https://proceedings.mlr.press/v306/marone26a.html.

Related Material