CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability

Xianzhen Luo, Jingyuan Zhang, Shiqi Zhou, Jinyang Huang, Chuan Xiao, Qingfu Zhu, Zhiyuan Ma, Yue Xing, Yang Yue, Wencong Zeng, Wanxiang Che
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:83137-83168, 2026.

Abstract

Evaluating and improving the security capabilities of code agents requires high-quality, executable vulnerability tasks. However, existing works rely on costly, unscalable manual reproduction and suffer from outdated data distributions. To address these, we present CVE-Factory, the first multi-agent framework to achieve expert-level quality in automatically transforming sparse CVE metadata into fully executable agentic tasks. Cross-validation against human expert reproductions shows that CVE-Factory achieves 95% solution correctness and 96% environment fidelity, confirming its expert-level quality. It is also evaluated on the latest realistic vulnerabilities and achieves a 66.2% verified success. This automation enables two downstream contributions. First, we construct LiveCVEBench, a continuously updated benchmark of 190 tasks spanning 14 languages and 153 repositories that captures emerging threats including AI-tooling vulnerabilities. Second, we synthesize over 1,000 executable training environments, the first large-scale scaling of agentic tasks in code security. Fine-tuned Qwen3-32B improves from 5.3% to 35.8% on LiveCVEBench, surpassing Claude 4.5 Sonnet, with gains generalizing to Terminal Bench (12.5% to 31.3%). We open-source all code, data, and models.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-luo26aj, title = {{CVE}-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability}, author = {Luo, Xianzhen and Zhang, Jingyuan and Zhou, Shiqi and Huang, Jinyang and Xiao, Chuan and Zhu, Qingfu and Ma, Zhiyuan and Xing, Yue and Yue, Yang and Zeng, Wencong and Che, Wanxiang}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {83137--83168}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/luo26aj/luo26aj.pdf}, url = {https://proceedings.mlr.press/v306/luo26aj.html}, abstract = {Evaluating and improving the security capabilities of code agents requires high-quality, executable vulnerability tasks. However, existing works rely on costly, unscalable manual reproduction and suffer from outdated data distributions. To address these, we present CVE-Factory, the first multi-agent framework to achieve expert-level quality in automatically transforming sparse CVE metadata into fully executable agentic tasks. Cross-validation against human expert reproductions shows that CVE-Factory achieves 95% solution correctness and 96% environment fidelity, confirming its expert-level quality. It is also evaluated on the latest realistic vulnerabilities and achieves a 66.2% verified success. This automation enables two downstream contributions. First, we construct LiveCVEBench, a continuously updated benchmark of 190 tasks spanning 14 languages and 153 repositories that captures emerging threats including AI-tooling vulnerabilities. Second, we synthesize over 1,000 executable training environments, the first large-scale scaling of agentic tasks in code security. Fine-tuned Qwen3-32B improves from 5.3% to 35.8% on LiveCVEBench, surpassing Claude 4.5 Sonnet, with gains generalizing to Terminal Bench (12.5% to 31.3%). We open-source all code, data, and models.} }
Endnote
%0 Conference Paper %T CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability %A Xianzhen Luo %A Jingyuan Zhang %A Shiqi Zhou %A Jinyang Huang %A Chuan Xiao %A Qingfu Zhu %A Zhiyuan Ma %A Yue Xing %A Yang Yue %A Wencong Zeng %A Wanxiang Che %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-luo26aj %I PMLR %P 83137--83168 %U https://proceedings.mlr.press/v306/luo26aj.html %V 306 %X Evaluating and improving the security capabilities of code agents requires high-quality, executable vulnerability tasks. However, existing works rely on costly, unscalable manual reproduction and suffer from outdated data distributions. To address these, we present CVE-Factory, the first multi-agent framework to achieve expert-level quality in automatically transforming sparse CVE metadata into fully executable agentic tasks. Cross-validation against human expert reproductions shows that CVE-Factory achieves 95% solution correctness and 96% environment fidelity, confirming its expert-level quality. It is also evaluated on the latest realistic vulnerabilities and achieves a 66.2% verified success. This automation enables two downstream contributions. First, we construct LiveCVEBench, a continuously updated benchmark of 190 tasks spanning 14 languages and 153 repositories that captures emerging threats including AI-tooling vulnerabilities. Second, we synthesize over 1,000 executable training environments, the first large-scale scaling of agentic tasks in code security. Fine-tuned Qwen3-32B improves from 5.3% to 35.8% on LiveCVEBench, surpassing Claude 4.5 Sonnet, with gains generalizing to Terminal Bench (12.5% to 31.3%). We open-source all code, data, and models.
APA
Luo, X., Zhang, J., Zhou, S., Huang, J., Xiao, C., Zhu, Q., Ma, Z., Xing, Y., Yue, Y., Zeng, W. & Che, W.. (2026). CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:83137-83168 Available from https://proceedings.mlr.press/v306/luo26aj.html.

Related Material