QiMeng-LibBench: Benchmarking LLM Agents for Library-Scale Cross-Architecture Migration

Weijia Li, Ke Gao, Jiajie Li, Han Sun, Yuhe Ding, Yongdong Mai, Yiran Le, Yongjie Qian, Zhibin Zhang, Xinyu Wang, Limin Cheng, Shouxu Kuang, Pengfei Chen, Ling Li
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:66903-66919, 2026.

Abstract

Cross-architecture migration of high-performance libraries dictates ecosystem readiness on emerging hardware. The challenge is twofold: disentangling library-scale dependencies and performance-critical kernels with ISA-specific SIMD intrinsics, often trading migration speed for peak performance. While LLM-based agents offer a promising approach, are confined to function-level tasks or scalar code, failing to assess agents’ capabilities and limitations in realistic, library-scale migration. We present QM-LibBench, a benchmark for cross-architecture library-scale code migration, featuring 85 critical kernels from widely used libraries, including OpenCV, libjpeg, and NCNN. It supports comprehensive evaluations of compilability, correctness, and performance across major transitions: ARM$\rightarrow$RISC-V, x86$\rightarrow$ARM, and ARM$\rightarrow$LoongArch. Evaluation of 12 SOTA agent-LLM combinations on QM-LibBench reveals that, due to the lack of library-level navigation and hardware-aware optimization, agents regress to superficial pattern matching, yielding only 20.88% correctness and 0.83 speedup for libjpeg. Motivated by these findings, we further propose FSCM, a multi-agent framework incorporating hardware-aware global reconfiguration and performance optimization. FSCM improves OpenCV correctness to 71%. The benchmark and code are available at https://github.com/WisdomJoy/QM-LibBench.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-li26ar, title = {{Q}i{M}eng-{L}ib{B}ench: Benchmarking {LLM} Agents for Library-Scale Cross-Architecture Migration}, author = {Li, Weijia and Gao, Ke and Li, Jiajie and Sun, Han and Ding, Yuhe and Mai, Yongdong and Le, Yiran and Qian, Yongjie and Zhang, Zhibin and Wang, Xinyu and Cheng, Limin and Kuang, Shouxu and Chen, Pengfei and Li, Ling}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {66903--66919}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/li26ar/li26ar.pdf}, url = {https://proceedings.mlr.press/v306/li26ar.html}, abstract = {Cross-architecture migration of high-performance libraries dictates ecosystem readiness on emerging hardware. The challenge is twofold: disentangling library-scale dependencies and performance-critical kernels with ISA-specific SIMD intrinsics, often trading migration speed for peak performance. While LLM-based agents offer a promising approach, are confined to function-level tasks or scalar code, failing to assess agents’ capabilities and limitations in realistic, library-scale migration. We present QM-LibBench, a benchmark for cross-architecture library-scale code migration, featuring 85 critical kernels from widely used libraries, including OpenCV, libjpeg, and NCNN. It supports comprehensive evaluations of compilability, correctness, and performance across major transitions: ARM$\rightarrow$RISC-V, x86$\rightarrow$ARM, and ARM$\rightarrow$LoongArch. Evaluation of 12 SOTA agent-LLM combinations on QM-LibBench reveals that, due to the lack of library-level navigation and hardware-aware optimization, agents regress to superficial pattern matching, yielding only 20.88% correctness and 0.83 speedup for libjpeg. Motivated by these findings, we further propose FSCM, a multi-agent framework incorporating hardware-aware global reconfiguration and performance optimization. FSCM improves OpenCV correctness to 71%. The benchmark and code are available at https://github.com/WisdomJoy/QM-LibBench.} }
Endnote
%0 Conference Paper %T QiMeng-LibBench: Benchmarking LLM Agents for Library-Scale Cross-Architecture Migration %A Weijia Li %A Ke Gao %A Jiajie Li %A Han Sun %A Yuhe Ding %A Yongdong Mai %A Yiran Le %A Yongjie Qian %A Zhibin Zhang %A Xinyu Wang %A Limin Cheng %A Shouxu Kuang %A Pengfei Chen %A Ling Li %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-li26ar %I PMLR %P 66903--66919 %U https://proceedings.mlr.press/v306/li26ar.html %V 306 %X Cross-architecture migration of high-performance libraries dictates ecosystem readiness on emerging hardware. The challenge is twofold: disentangling library-scale dependencies and performance-critical kernels with ISA-specific SIMD intrinsics, often trading migration speed for peak performance. While LLM-based agents offer a promising approach, are confined to function-level tasks or scalar code, failing to assess agents’ capabilities and limitations in realistic, library-scale migration. We present QM-LibBench, a benchmark for cross-architecture library-scale code migration, featuring 85 critical kernels from widely used libraries, including OpenCV, libjpeg, and NCNN. It supports comprehensive evaluations of compilability, correctness, and performance across major transitions: ARM$\rightarrow$RISC-V, x86$\rightarrow$ARM, and ARM$\rightarrow$LoongArch. Evaluation of 12 SOTA agent-LLM combinations on QM-LibBench reveals that, due to the lack of library-level navigation and hardware-aware optimization, agents regress to superficial pattern matching, yielding only 20.88% correctness and 0.83 speedup for libjpeg. Motivated by these findings, we further propose FSCM, a multi-agent framework incorporating hardware-aware global reconfiguration and performance optimization. FSCM improves OpenCV correctness to 71%. The benchmark and code are available at https://github.com/WisdomJoy/QM-LibBench.
APA
Li, W., Gao, K., Li, J., Sun, H., Ding, Y., Mai, Y., Le, Y., Qian, Y., Zhang, Z., Wang, X., Cheng, L., Kuang, S., Chen, P. & Li, L.. (2026). QiMeng-LibBench: Benchmarking LLM Agents for Library-Scale Cross-Architecture Migration. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:66903-66919 Available from https://proceedings.mlr.press/v306/li26ar.html.

Related Material