SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models

Jingxuan Xu, Ken Deng, Weihao Li, Songwei Yu, Haoyang Huang, Xinping Lei, Yifan Yao, Huaixi Tang, Zhiyi Lai, Kepeng Lei, Zizheng Zhan, Yanan Wu, Chenchen Zhang, Wenqiang Zhu, Wen Xiang, Zongxian Feng, Han Li, Junqi Xiong, Dailin Li, Zuchen Gao, Kun Wu, Yuanxing Zhang, Wuxuan Gong, Ziyuan Gao, Guanxiang Wang, Yirong Xue, Mengfei Xie, Xiaojiang Zhang, Jinghui Wang, Wenhao Zhuang, Zheng Lin, Huiming Wang, Zhaoxiang Zhang, Yuqun Zhang, Haotian Zhang, Ming Sun, Chen Bin, Jiaheng Liu
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:140878-140902, 2026.

Abstract

Evaluating large language models (LLMs) for software engineering has been limited by narrow task coverage, language bias, and insufficient alignment with real-world developer workflows. Existing benchmarks often focus on algorithmic problems or Python-centric bug fixing, leaving critical dimensions of software engineering underexplored. To address these gaps, we introduce SWE-Compass, a comprehensive benchmark that unifies heterogeneous code-related evaluations into a structured and production-aligned framework. SWE-Compass spans 8 task types, 8 programming scenarios, and 10 programming languages, with 2000 high-quality instances curated from authentic GitHub pull requests and refined through systematic filtering and validation. We benchmark ten state-of-the-art LLMs under two agentic frameworks, SWE-Agent and Claude Code, revealing a clear hierarchy of difficulty across task types, languages, and scenarios. Moreover, by aligning evaluation with real-world developer practices, we hope SWE-Compass can provide a rigorous and reproducible foundation for diagnosing and advancing agentic coding capabilities in large language models.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-xu26k, title = {{SWE}-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models}, author = {Xu, Jingxuan and Deng, Ken and Li, Weihao and Yu, Songwei and Huang, Haoyang and Lei, Xinping and Yao, Yifan and Tang, Huaixi and Lai, Zhiyi and Lei, Kepeng and Zhan, Zizheng and Wu, Yanan and Zhang, Chenchen and Zhu, Wenqiang and Xiang, Wen and Feng, Zongxian and Li, Han and Xiong, Junqi and Li, Dailin and Gao, Zuchen and Wu, Kun and Zhang, Yuanxing and Gong, Wuxuan and Gao, Ziyuan and Wang, Guanxiang and Xue, Yirong and Xie, Mengfei and Zhang, Xiaojiang and Wang, Jinghui and Zhuang, Wenhao and Lin, Zheng and Wang, Huiming and Zhang, Zhaoxiang and Zhang, Yuqun and Zhang, Haotian and Sun, Ming and Bin, Chen and Liu, Jiaheng}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {140878--140902}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/xu26k/xu26k.pdf}, url = {https://proceedings.mlr.press/v306/xu26k.html}, abstract = {Evaluating large language models (LLMs) for software engineering has been limited by narrow task coverage, language bias, and insufficient alignment with real-world developer workflows. Existing benchmarks often focus on algorithmic problems or Python-centric bug fixing, leaving critical dimensions of software engineering underexplored. To address these gaps, we introduce SWE-Compass, a comprehensive benchmark that unifies heterogeneous code-related evaluations into a structured and production-aligned framework. SWE-Compass spans 8 task types, 8 programming scenarios, and 10 programming languages, with 2000 high-quality instances curated from authentic GitHub pull requests and refined through systematic filtering and validation. We benchmark ten state-of-the-art LLMs under two agentic frameworks, SWE-Agent and Claude Code, revealing a clear hierarchy of difficulty across task types, languages, and scenarios. Moreover, by aligning evaluation with real-world developer practices, we hope SWE-Compass can provide a rigorous and reproducible foundation for diagnosing and advancing agentic coding capabilities in large language models.} }
Endnote
%0 Conference Paper %T SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models %A Jingxuan Xu %A Ken Deng %A Weihao Li %A Songwei Yu %A Haoyang Huang %A Xinping Lei %A Yifan Yao %A Huaixi Tang %A Zhiyi Lai %A Kepeng Lei %A Zizheng Zhan %A Yanan Wu %A Chenchen Zhang %A Wenqiang Zhu %A Wen Xiang %A Zongxian Feng %A Han Li %A Junqi Xiong %A Dailin Li %A Zuchen Gao %A Kun Wu %A Yuanxing Zhang %A Wuxuan Gong %A Ziyuan Gao %A Guanxiang Wang %A Yirong Xue %A Mengfei Xie %A Xiaojiang Zhang %A Jinghui Wang %A Wenhao Zhuang %A Zheng Lin %A Huiming Wang %A Zhaoxiang Zhang %A Yuqun Zhang %A Haotian Zhang %A Ming Sun %A Chen Bin %A Jiaheng Liu %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-xu26k %I PMLR %P 140878--140902 %U https://proceedings.mlr.press/v306/xu26k.html %V 306 %X Evaluating large language models (LLMs) for software engineering has been limited by narrow task coverage, language bias, and insufficient alignment with real-world developer workflows. Existing benchmarks often focus on algorithmic problems or Python-centric bug fixing, leaving critical dimensions of software engineering underexplored. To address these gaps, we introduce SWE-Compass, a comprehensive benchmark that unifies heterogeneous code-related evaluations into a structured and production-aligned framework. SWE-Compass spans 8 task types, 8 programming scenarios, and 10 programming languages, with 2000 high-quality instances curated from authentic GitHub pull requests and refined through systematic filtering and validation. We benchmark ten state-of-the-art LLMs under two agentic frameworks, SWE-Agent and Claude Code, revealing a clear hierarchy of difficulty across task types, languages, and scenarios. Moreover, by aligning evaluation with real-world developer practices, we hope SWE-Compass can provide a rigorous and reproducible foundation for diagnosing and advancing agentic coding capabilities in large language models.
APA
Xu, J., Deng, K., Li, W., Yu, S., Huang, H., Lei, X., Yao, Y., Tang, H., Lai, Z., Lei, K., Zhan, Z., Wu, Y., Zhang, C., Zhu, W., Xiang, W., Feng, Z., Li, H., Xiong, J., Li, D., Gao, Z., Wu, K., Zhang, Y., Gong, W., Gao, Z., Wang, G., Xue, Y., Xie, M., Zhang, X., Wang, J., Zhuang, W., Lin, Z., Wang, H., Zhang, Z., Zhang, Y., Zhang, H., Sun, M., Bin, C. & Liu, J.. (2026). SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:140878-140902 Available from https://proceedings.mlr.press/v306/xu26k.html.

Related Material