MORE: A Multilingual Document Parsing Benchmark and Evaluation

Long Xu, Binghong Wu, Tinghao Yu, Hao Feng, Zhenyu Huang, Haoqing Jiang, Yunhao Wang, Shuo Huang, Feng Zhang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:142324-142345, 2026.

Abstract

Multilingual documents encapsulate rich regional cultures, scientific discoveries, and historical records. Parsing this content into structured, machine-readable formats is critical for unlocking global knowledge. However, existing benchmarks predominantly focus on high-resource languages like English and Chinese, creating an $\textit{evaluation blind spot}$ concerning model performance on other languages. While recent Vision-Language Models (VLMs) claim support for hundreds of languages, the lack of ground truth makes it impossible to empirically verify these capabilities. To bridge this gap, we introduce $\textbf{MORE}$, a large-scale benchmark designed for multilingual document parsing evaluation. MORE distinguishes itself through three key dimensions: (1) $\textbf{Unprecedented Scale}$: It covers $\textbf{149 languages}$, making it the most linguistically diverse benchmark to date; (2) $\textbf{Structural Complexity}$: Unlike previous works, it extends evaluation beyond plain text to include structural elements such as code blocks, tables, and catalogs; and (3) $\textbf{Data Authenticity}$: All samples are curated from real-world documents via a model-assisted, human-refined annotation pipeline. We evaluate state-of-the-art models using MORE, establishing new performance baselines for long-tail languages and validating the benchmark’s effectiveness in diagnosing model capabilities in realistic, diverse scenarios. The MORE dataset will be available at https://github.com/zimoqingfeng/MORE.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-xu26bs, title = {{MORE}: A Multilingual Document Parsing Benchmark and Evaluation}, author = {Xu, Long and Wu, Binghong and Yu, Tinghao and Feng, Hao and Huang, Zhenyu and Jiang, Haoqing and Wang, Yunhao and Huang, Shuo and Zhang, Feng}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {142324--142345}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/xu26bs/xu26bs.pdf}, url = {https://proceedings.mlr.press/v306/xu26bs.html}, abstract = {Multilingual documents encapsulate rich regional cultures, scientific discoveries, and historical records. Parsing this content into structured, machine-readable formats is critical for unlocking global knowledge. However, existing benchmarks predominantly focus on high-resource languages like English and Chinese, creating an $\textit{evaluation blind spot}$ concerning model performance on other languages. While recent Vision-Language Models (VLMs) claim support for hundreds of languages, the lack of ground truth makes it impossible to empirically verify these capabilities. To bridge this gap, we introduce $\textbf{MORE}$, a large-scale benchmark designed for multilingual document parsing evaluation. MORE distinguishes itself through three key dimensions: (1) $\textbf{Unprecedented Scale}$: It covers $\textbf{149 languages}$, making it the most linguistically diverse benchmark to date; (2) $\textbf{Structural Complexity}$: Unlike previous works, it extends evaluation beyond plain text to include structural elements such as code blocks, tables, and catalogs; and (3) $\textbf{Data Authenticity}$: All samples are curated from real-world documents via a model-assisted, human-refined annotation pipeline. We evaluate state-of-the-art models using MORE, establishing new performance baselines for long-tail languages and validating the benchmark’s effectiveness in diagnosing model capabilities in realistic, diverse scenarios. The MORE dataset will be available at https://github.com/zimoqingfeng/MORE.} }
Endnote
%0 Conference Paper %T MORE: A Multilingual Document Parsing Benchmark and Evaluation %A Long Xu %A Binghong Wu %A Tinghao Yu %A Hao Feng %A Zhenyu Huang %A Haoqing Jiang %A Yunhao Wang %A Shuo Huang %A Feng Zhang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-xu26bs %I PMLR %P 142324--142345 %U https://proceedings.mlr.press/v306/xu26bs.html %V 306 %X Multilingual documents encapsulate rich regional cultures, scientific discoveries, and historical records. Parsing this content into structured, machine-readable formats is critical for unlocking global knowledge. However, existing benchmarks predominantly focus on high-resource languages like English and Chinese, creating an $\textit{evaluation blind spot}$ concerning model performance on other languages. While recent Vision-Language Models (VLMs) claim support for hundreds of languages, the lack of ground truth makes it impossible to empirically verify these capabilities. To bridge this gap, we introduce $\textbf{MORE}$, a large-scale benchmark designed for multilingual document parsing evaluation. MORE distinguishes itself through three key dimensions: (1) $\textbf{Unprecedented Scale}$: It covers $\textbf{149 languages}$, making it the most linguistically diverse benchmark to date; (2) $\textbf{Structural Complexity}$: Unlike previous works, it extends evaluation beyond plain text to include structural elements such as code blocks, tables, and catalogs; and (3) $\textbf{Data Authenticity}$: All samples are curated from real-world documents via a model-assisted, human-refined annotation pipeline. We evaluate state-of-the-art models using MORE, establishing new performance baselines for long-tail languages and validating the benchmark’s effectiveness in diagnosing model capabilities in realistic, diverse scenarios. The MORE dataset will be available at https://github.com/zimoqingfeng/MORE.
APA
Xu, L., Wu, B., Yu, T., Feng, H., Huang, Z., Jiang, H., Wang, Y., Huang, S. & Zhang, F.. (2026). MORE: A Multilingual Document Parsing Benchmark and Evaluation. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:142324-142345 Available from https://proceedings.mlr.press/v306/xu26bs.html.

Related Material