On Path to Multimodal Historical Reasoning: HistBench and HistAgent

Jiahao Qiu, Fulian Xiao, Yimin Wang, Yuchen Mao, Yijia Chen, Xinzhe Juan, Siran Wang, Xuan Qi, Tongcheng Zhang, Zixin Yao, Jiacheng Guo, Yifu Lu, Charles Argon, Jundi Cui, Daixin Chen, Junran Zhou, Shuyao Zhou, Zhanpeng Zhou, Ling Yang, Shilong Liu, Hongru Wang, Kaixuan Huang, Xun Jiang, Xi Gao, Mengdi Wang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:102312-102338, 2026.

Abstract

Recent advances in large language models (LLMs) have led to remarkable progress across various domains, yet their capabilities in the humanities, particularly history, remain underexplored. Historical reasoning poses unique challenges for LLMs, involving multimodal source interpretation, temporal inference, and cross-linguistic analysis. Existing general-purpose agents perform well on many current benchmarks but lack the domain expertise needed to address complex historical questions. To address this gap, we introduce HistBench, a new benchmark of 414 high-quality and carefully-reviewed questions stratified by difficulty and designed to evaluate LLM’s capacity for historical reasoning. The tasks span a wide range of historical problems—from factual retrieval based on primary sources to interpretive analysis of manuscripts and images, to interdisciplinary challenges involving archaeology, linguistics, or cultural history. Furthermore, the benchmark dataset spans 29 ancient and modern languages and covers a wide range of historical periods and world regions. Finding the poor performance of LLMs and other agents on HistBench, we further present HistAgent, a history-specific agent equipped with carefully designed tools for OCR, translation, archival search, and image understanding in history. On HistBench, HistAgent based on GPT-4o achieves an accuracy of 28.50% pass@1 and 36.47% pass@2, significantly outperforming LLMs with online search and generalist agents, including GPT-4o (18.60%), DeepSeek-R1 (14.98%), Grok 3 (17.63%) and Open Deep Research by smolagents (19.57% pass@1 and 25.12% pass@2). These results highlight the limitations of existing LLMs and generalist agents and demonstrate the advantages of HistAgent for historical reasoning. Notably, HistAgent also achieves 60.00% pass@1 accuracy on the GAIA benchmark, showing that domain-specific customization doesn’t hinder HistAgent’s competitive performance on real-world general tasks. Code is available at https://github.com/CharlesQ9/HistAgent.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-qiu26s, title = {On Path to Multimodal Historical Reasoning: {H}ist{B}ench and {H}ist{A}gent}, author = {Qiu, Jiahao and Xiao, Fulian and Wang, Yimin and Mao, Yuchen and Chen, Yijia and Juan, Xinzhe and Wang, Siran and Qi, Xuan and Zhang, Tongcheng and Yao, Zixin and Guo, Jiacheng and Lu, Yifu and Argon, Charles and Cui, Jundi and Chen, Daixin and Zhou, Junran and Zhou, Shuyao and Zhou, Zhanpeng and Yang, Ling and Liu, Shilong and Wang, Hongru and Huang, Kaixuan and Jiang, Xun and Gao, Xi and Wang, Mengdi}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {102312--102338}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/qiu26s/qiu26s.pdf}, url = {https://proceedings.mlr.press/v306/qiu26s.html}, abstract = {Recent advances in large language models (LLMs) have led to remarkable progress across various domains, yet their capabilities in the humanities, particularly history, remain underexplored. Historical reasoning poses unique challenges for LLMs, involving multimodal source interpretation, temporal inference, and cross-linguistic analysis. Existing general-purpose agents perform well on many current benchmarks but lack the domain expertise needed to address complex historical questions. To address this gap, we introduce HistBench, a new benchmark of 414 high-quality and carefully-reviewed questions stratified by difficulty and designed to evaluate LLM’s capacity for historical reasoning. The tasks span a wide range of historical problems—from factual retrieval based on primary sources to interpretive analysis of manuscripts and images, to interdisciplinary challenges involving archaeology, linguistics, or cultural history. Furthermore, the benchmark dataset spans 29 ancient and modern languages and covers a wide range of historical periods and world regions. Finding the poor performance of LLMs and other agents on HistBench, we further present HistAgent, a history-specific agent equipped with carefully designed tools for OCR, translation, archival search, and image understanding in history. On HistBench, HistAgent based on GPT-4o achieves an accuracy of 28.50% pass@1 and 36.47% pass@2, significantly outperforming LLMs with online search and generalist agents, including GPT-4o (18.60%), DeepSeek-R1 (14.98%), Grok 3 (17.63%) and Open Deep Research by smolagents (19.57% pass@1 and 25.12% pass@2). These results highlight the limitations of existing LLMs and generalist agents and demonstrate the advantages of HistAgent for historical reasoning. Notably, HistAgent also achieves 60.00% pass@1 accuracy on the GAIA benchmark, showing that domain-specific customization doesn’t hinder HistAgent’s competitive performance on real-world general tasks. Code is available at https://github.com/CharlesQ9/HistAgent.} }
Endnote
%0 Conference Paper %T On Path to Multimodal Historical Reasoning: HistBench and HistAgent %A Jiahao Qiu %A Fulian Xiao %A Yimin Wang %A Yuchen Mao %A Yijia Chen %A Xinzhe Juan %A Siran Wang %A Xuan Qi %A Tongcheng Zhang %A Zixin Yao %A Jiacheng Guo %A Yifu Lu %A Charles Argon %A Jundi Cui %A Daixin Chen %A Junran Zhou %A Shuyao Zhou %A Zhanpeng Zhou %A Ling Yang %A Shilong Liu %A Hongru Wang %A Kaixuan Huang %A Xun Jiang %A Xi Gao %A Mengdi Wang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-qiu26s %I PMLR %P 102312--102338 %U https://proceedings.mlr.press/v306/qiu26s.html %V 306 %X Recent advances in large language models (LLMs) have led to remarkable progress across various domains, yet their capabilities in the humanities, particularly history, remain underexplored. Historical reasoning poses unique challenges for LLMs, involving multimodal source interpretation, temporal inference, and cross-linguistic analysis. Existing general-purpose agents perform well on many current benchmarks but lack the domain expertise needed to address complex historical questions. To address this gap, we introduce HistBench, a new benchmark of 414 high-quality and carefully-reviewed questions stratified by difficulty and designed to evaluate LLM’s capacity for historical reasoning. The tasks span a wide range of historical problems—from factual retrieval based on primary sources to interpretive analysis of manuscripts and images, to interdisciplinary challenges involving archaeology, linguistics, or cultural history. Furthermore, the benchmark dataset spans 29 ancient and modern languages and covers a wide range of historical periods and world regions. Finding the poor performance of LLMs and other agents on HistBench, we further present HistAgent, a history-specific agent equipped with carefully designed tools for OCR, translation, archival search, and image understanding in history. On HistBench, HistAgent based on GPT-4o achieves an accuracy of 28.50% pass@1 and 36.47% pass@2, significantly outperforming LLMs with online search and generalist agents, including GPT-4o (18.60%), DeepSeek-R1 (14.98%), Grok 3 (17.63%) and Open Deep Research by smolagents (19.57% pass@1 and 25.12% pass@2). These results highlight the limitations of existing LLMs and generalist agents and demonstrate the advantages of HistAgent for historical reasoning. Notably, HistAgent also achieves 60.00% pass@1 accuracy on the GAIA benchmark, showing that domain-specific customization doesn’t hinder HistAgent’s competitive performance on real-world general tasks. Code is available at https://github.com/CharlesQ9/HistAgent.
APA
Qiu, J., Xiao, F., Wang, Y., Mao, Y., Chen, Y., Juan, X., Wang, S., Qi, X., Zhang, T., Yao, Z., Guo, J., Lu, Y., Argon, C., Cui, J., Chen, D., Zhou, J., Zhou, S., Zhou, Z., Yang, L., Liu, S., Wang, H., Huang, K., Jiang, X., Gao, X. & Wang, M.. (2026). On Path to Multimodal Historical Reasoning: HistBench and HistAgent. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:102312-102338 Available from https://proceedings.mlr.press/v306/qiu26s.html.

Related Material