Lesan – Machine Translation for Low Resource Languages

Asmelash Teka Hadgu, Abel Aregawi, Adam Beaudoin
Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track, PMLR 176:297-301, 2022.

Abstract

Millions of people around the world can not access content on the Web because most of the content is not readily available in their language. Machine translation (MT) systems have the potential to change this for many languages. Current MT systems provide very accurate results for high resource language pairs, e.g., German and English. However, for many low resource languages, MT is still under active research. The key challenge is lack of datasets to build these systems. We present Lesan (https://lesan.ai/), an MT system for low resource languages. Our pipeline solves the key bottleneck to low resource MT by leveraging online and offline sources, a custom Optical Character Recognition (OCR) system for Ethiopic and an automatic alignment module. The final step in the pipeline is a sequence to sequence model that takes parallel corpus as input and gives us a translation model. Lesan{’}s translation model is based on the Transformer architecture. After constructing a base model, back translation is used to leverage monolingual corpora. Currently Lesan supports translation to and from Tigrinya, Amharic and English. We perform extensive human evaluation and show that Lesan outperforms state-of-the-art systems such as Google Translate and Microsoft Translator across all six pairs. Lesan is freely available and has served more than 10 million translations so far. At the moment, there are only 217 Tigrinya and 15,009 Amharic Wikipedia articles. We believe that Lesan will contribute towards democratizing access to the Web through MT for millions of people.

Cite this Paper

BibTeX
@InProceedings{pmlr-v176-hadgu22a, title = {Lesan {–} Machine Translation for Low Resource Languages}, author = {Hadgu, Asmelash Teka and Aregawi, Abel and Beaudoin, Adam}, booktitle = {Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track}, pages = {297--301}, year = {2022}, editor = {Kiela, Douwe and Ciccone, Marco and Caputo, Barbara}, volume = {176}, series = {Proceedings of Machine Learning Research}, month = {06--14 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v176/hadgu22a/hadgu22a.pdf}, url = {https://proceedings.mlr.press/v176/hadgu22a.html}, abstract = {Millions of people around the world can not access content on the Web because most of the content is not readily available in their language. Machine translation (MT) systems have the potential to change this for many languages. Current MT systems provide very accurate results for high resource language pairs, e.g., German and English. However, for many low resource languages, MT is still under active research. The key challenge is lack of datasets to build these systems. We present Lesan (https://lesan.ai/), an MT system for low resource languages. Our pipeline solves the key bottleneck to low resource MT by leveraging online and offline sources, a custom Optical Character Recognition (OCR) system for Ethiopic and an automatic alignment module. The final step in the pipeline is a sequence to sequence model that takes parallel corpus as input and gives us a translation model. Lesan{’}s translation model is based on the Transformer architecture. After constructing a base model, back translation is used to leverage monolingual corpora. Currently Lesan supports translation to and from Tigrinya, Amharic and English. We perform extensive human evaluation and show that Lesan outperforms state-of-the-art systems such as Google Translate and Microsoft Translator across all six pairs. Lesan is freely available and has served more than 10 million translations so far. At the moment, there are only 217 Tigrinya and 15,009 Amharic Wikipedia articles. We believe that Lesan will contribute towards democratizing access to the Web through MT for millions of people.} }
Endnote
%0 Conference Paper %T Lesan – Machine Translation for Low Resource Languages %A Asmelash Teka Hadgu %A Abel Aregawi %A Adam Beaudoin %B Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track %C Proceedings of Machine Learning Research %D 2022 %E Douwe Kiela %E Marco Ciccone %E Barbara Caputo %F pmlr-v176-hadgu22a %I PMLR %P 297--301 %U https://proceedings.mlr.press/v176/hadgu22a.html %V 176 %X Millions of people around the world can not access content on the Web because most of the content is not readily available in their language. Machine translation (MT) systems have the potential to change this for many languages. Current MT systems provide very accurate results for high resource language pairs, e.g., German and English. However, for many low resource languages, MT is still under active research. The key challenge is lack of datasets to build these systems. We present Lesan (https://lesan.ai/), an MT system for low resource languages. Our pipeline solves the key bottleneck to low resource MT by leveraging online and offline sources, a custom Optical Character Recognition (OCR) system for Ethiopic and an automatic alignment module. The final step in the pipeline is a sequence to sequence model that takes parallel corpus as input and gives us a translation model. Lesan{’}s translation model is based on the Transformer architecture. After constructing a base model, back translation is used to leverage monolingual corpora. Currently Lesan supports translation to and from Tigrinya, Amharic and English. We perform extensive human evaluation and show that Lesan outperforms state-of-the-art systems such as Google Translate and Microsoft Translator across all six pairs. Lesan is freely available and has served more than 10 million translations so far. At the moment, there are only 217 Tigrinya and 15,009 Amharic Wikipedia articles. We believe that Lesan will contribute towards democratizing access to the Web through MT for millions of people.
APA
Hadgu, A.T., Aregawi, A. & Beaudoin, A.. (2022). Lesan – Machine Translation for Low Resource Languages. Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track, in Proceedings of Machine Learning Research 176:297-301 Available from https://proceedings.mlr.press/v176/hadgu22a.html.

Related Material