Combatting Language Forgetting in Low-Resourced Settings

Emmanuel Rassou
DLI 2025 Research Track, PMLR 302:1-11, 2026.

Abstract

Neural machine translation becomes a continual learning challenge as language evolves over time. While Transformer-based models excel at capturing linguistic patterns from large corpora, they require continual updates to adapt to new language use without losing previously acquired knowledge. In this work, we introduce Latent Replay Buffers to the NLP domain for the first time by implementing and fine-tuning our Latent Replay Transformer. We conduct initial experiments for low-resource languages on Small-100, a distilled version of a multilingual transformer trained on 100 languages, to be well-suited for deployment in memory- and data-constrained environments. Our findings reveal an intriguing trade-off in the selection of latent activations to store for effective replay. We release our code to support both the Continual Learning and NLP for Low-Resourced Languages communities. Keywords: Continual Learning, Transformers, Low-Resourced Languages.

Cite this Paper

BibTeX
@InProceedings{pmlr-v302-rassou26a, title = {Combatting Language Forgetting in Low-Resourced Settings}, author = {Rassou, Emmanuel}, booktitle = {DLI 2025 Research Track}, pages = {1--11}, year = {2026}, editor = {Haddad, Hatem and Kahira, Albert Njoroge and Bourhim, Sofia and Olatunji, Iyiola Emmanuel and Makhafola, Lesego and Mwase, Christine}, volume = {302}, series = {Proceedings of Machine Learning Research}, month = {17--22 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v302/main/assets/rassou26a/rassou26a.pdf}, url = {https://proceedings.mlr.press/v302/rassou26a.html}, abstract = {Neural machine translation becomes a continual learning challenge as language evolves over time. While Transformer-based models excel at capturing linguistic patterns from large corpora, they require continual updates to adapt to new language use without losing previously acquired knowledge. In this work, we introduce Latent Replay Buffers to the NLP domain for the first time by implementing and fine-tuning our Latent Replay Transformer. We conduct initial experiments for low-resource languages on Small-100, a distilled version of a multilingual transformer trained on 100 languages, to be well-suited for deployment in memory- and data-constrained environments. Our findings reveal an intriguing trade-off in the selection of latent activations to store for effective replay. We release our code to support both the Continual Learning and NLP for Low-Resourced Languages communities. Keywords: Continual Learning, Transformers, Low-Resourced Languages.} }
Endnote
%0 Conference Paper %T Combatting Language Forgetting in Low-Resourced Settings %A Emmanuel Rassou %B DLI 2025 Research Track %C Proceedings of Machine Learning Research %D 2026 %E Hatem Haddad %E Albert Njoroge Kahira %E Sofia Bourhim %E Iyiola Emmanuel Olatunji %E Lesego Makhafola %E Christine Mwase %F pmlr-v302-rassou26a %I PMLR %P 1--11 %U https://proceedings.mlr.press/v302/rassou26a.html %V 302 %X Neural machine translation becomes a continual learning challenge as language evolves over time. While Transformer-based models excel at capturing linguistic patterns from large corpora, they require continual updates to adapt to new language use without losing previously acquired knowledge. In this work, we introduce Latent Replay Buffers to the NLP domain for the first time by implementing and fine-tuning our Latent Replay Transformer. We conduct initial experiments for low-resource languages on Small-100, a distilled version of a multilingual transformer trained on 100 languages, to be well-suited for deployment in memory- and data-constrained environments. Our findings reveal an intriguing trade-off in the selection of latent activations to store for effective replay. We release our code to support both the Continual Learning and NLP for Low-Resourced Languages communities. Keywords: Continual Learning, Transformers, Low-Resourced Languages.
APA
Rassou, E.. (2026). Combatting Language Forgetting in Low-Resourced Settings. DLI 2025 Research Track, in Proceedings of Machine Learning Research 302:1-11 Available from https://proceedings.mlr.press/v302/rassou26a.html.

Related Material