TestExplora: Benchmarking LLMs for Proactive Bug Discovery via Repository-Level Test Generation

Steven Liu, Jane Luo, Xin Zhang, Aofan Liu, Hao Liu, Jie Wu, Ziyang Huang, Yangyu Huang, Yu Kang, Scarlett Li
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:77728-77771, 2026.

Abstract

Given that Large Language Models (LLMs) are increasingly applied to automate software development, comprehensive software assurance spans three distinct goals: regression prevention, reactive reproduction, and proactive discovery. Current evaluations systematically overlook the third goal. Specifically, they either constrain models to a compliance trap by treating existing code as the ground truth for regression prevention, or rely on post-failure artifacts (e.g., issue reports) for reactive bug reproduction, failing to expose defects before they manifest as failures. To bridge this gap, we present TestExplora, a benchmark designed to evaluate LLMs as proactive testers within full-scale, realistic repository environments. Comprising 2,389 tasks across 482 repositories, TestExplora conceals all defect-related information, forcing models to uncover bugs by identifying discrepancies between implementation and documentation-derived intent—utilizing documentation as the reference oracle. Furthermore, to ensure sustainable evaluation and mitigate risks of data leakage in static datasets, we propose a continuous, time-aware data collection framework. Our evaluation reveals a significant capability gap: state-of-the-art models achieve a maximum Fail-to-Pass ($F2P$) rate of only 16.06%. Further analysis indicates that navigating complex cross-module interactions and leveraging agentic exploration are critical to advancing LLMs toward autonomous software quality assurance. Consistent with this, SWEAgent instantiated with GPT-5-mini achieves an $F2P$ of 17.27% and an $F2P@5$ of 29.7%, highlighting the effectiveness and promise of agentic exploration in proactive bug discovery tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-liu26ct, title = {{T}est{E}xplora: Benchmarking {LLM}s for Proactive Bug Discovery via Repository-Level Test Generation}, author = {Liu, Steven and Luo, Jane and Zhang, Xin and Liu, Aofan and Liu, Hao and Wu, Jie and Huang, Ziyang and Huang, Yangyu and Kang, Yu and Li, Scarlett}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {77728--77771}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/liu26ct/liu26ct.pdf}, url = {https://proceedings.mlr.press/v306/liu26ct.html}, abstract = {Given that Large Language Models (LLMs) are increasingly applied to automate software development, comprehensive software assurance spans three distinct goals: regression prevention, reactive reproduction, and proactive discovery. Current evaluations systematically overlook the third goal. Specifically, they either constrain models to a compliance trap by treating existing code as the ground truth for regression prevention, or rely on post-failure artifacts (e.g., issue reports) for reactive bug reproduction, failing to expose defects before they manifest as failures. To bridge this gap, we present TestExplora, a benchmark designed to evaluate LLMs as proactive testers within full-scale, realistic repository environments. Comprising 2,389 tasks across 482 repositories, TestExplora conceals all defect-related information, forcing models to uncover bugs by identifying discrepancies between implementation and documentation-derived intent—utilizing documentation as the reference oracle. Furthermore, to ensure sustainable evaluation and mitigate risks of data leakage in static datasets, we propose a continuous, time-aware data collection framework. Our evaluation reveals a significant capability gap: state-of-the-art models achieve a maximum Fail-to-Pass ($F2P$) rate of only 16.06%. Further analysis indicates that navigating complex cross-module interactions and leveraging agentic exploration are critical to advancing LLMs toward autonomous software quality assurance. Consistent with this, SWEAgent instantiated with GPT-5-mini achieves an $F2P$ of 17.27% and an $F2P@5$ of 29.7%, highlighting the effectiveness and promise of agentic exploration in proactive bug discovery tasks.} }
Endnote
%0 Conference Paper %T TestExplora: Benchmarking LLMs for Proactive Bug Discovery via Repository-Level Test Generation %A Steven Liu %A Jane Luo %A Xin Zhang %A Aofan Liu %A Hao Liu %A Jie Wu %A Ziyang Huang %A Yangyu Huang %A Yu Kang %A Scarlett Li %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-liu26ct %I PMLR %P 77728--77771 %U https://proceedings.mlr.press/v306/liu26ct.html %V 306 %X Given that Large Language Models (LLMs) are increasingly applied to automate software development, comprehensive software assurance spans three distinct goals: regression prevention, reactive reproduction, and proactive discovery. Current evaluations systematically overlook the third goal. Specifically, they either constrain models to a compliance trap by treating existing code as the ground truth for regression prevention, or rely on post-failure artifacts (e.g., issue reports) for reactive bug reproduction, failing to expose defects before they manifest as failures. To bridge this gap, we present TestExplora, a benchmark designed to evaluate LLMs as proactive testers within full-scale, realistic repository environments. Comprising 2,389 tasks across 482 repositories, TestExplora conceals all defect-related information, forcing models to uncover bugs by identifying discrepancies between implementation and documentation-derived intent—utilizing documentation as the reference oracle. Furthermore, to ensure sustainable evaluation and mitigate risks of data leakage in static datasets, we propose a continuous, time-aware data collection framework. Our evaluation reveals a significant capability gap: state-of-the-art models achieve a maximum Fail-to-Pass ($F2P$) rate of only 16.06%. Further analysis indicates that navigating complex cross-module interactions and leveraging agentic exploration are critical to advancing LLMs toward autonomous software quality assurance. Consistent with this, SWEAgent instantiated with GPT-5-mini achieves an $F2P$ of 17.27% and an $F2P@5$ of 29.7%, highlighting the effectiveness and promise of agentic exploration in proactive bug discovery tasks.
APA
Liu, S., Luo, J., Zhang, X., Liu, A., Liu, H., Wu, J., Huang, Z., Huang, Y., Kang, Y. & Li, S.. (2026). TestExplora: Benchmarking LLMs for Proactive Bug Discovery via Repository-Level Test Generation. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:77728-77771 Available from https://proceedings.mlr.press/v306/liu26ct.html.

Related Material