SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark

Boxi Yu, Yang Cao, Yuzhong Zhang, Liting Lin, Junjielong Xu, Zhiqing Zhong, Qinghua Xu, Guancheng Wang, Jialun Cao, Shing-Chi Cheung, Pinjia He, Lionel Briand
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:149974-150011, 2026.

Abstract

The SWE-Bench Verified leaderboard is approaching saturation, with the top system achieving 78.80%. However, we reveal that this performance is inflated: our re-evaluation demonstrates that one in five “solved” patches from the top-30 agents are semantically incorrect, passing only because weak test suites fail to expose their errors. We present SWE-ABS, an adversarial framework that strengthens test suites through a two-stage pipeline: (1) coverage-driven augmentation utilizing program slicing to target untested code regions, and (2) mutation-driven adversarial testing that synthesizes plausible-but-incorrect patches to expose semantic blind spots. On SWE-Bench Verified (500 instances), SWE-ABS strengthens 50.2% of instances (a $25.1\times$ improvement over prior work) and rejects 19.78% of previously passing patches. Consequently, the top agent’s score decreases from 78.80% to 62.20%, causing significant leaderboard reshuffling (e.g., the top-ranked agent drops to 5th place).

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-yu26b, title = {{SWE}-{ABS}: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark}, author = {Yu, Boxi and Cao, Yang and Zhang, Yuzhong and Lin, Liting and Xu, Junjielong and Zhong, Zhiqing and Xu, Qinghua and Wang, Guancheng and Cao, Jialun and Cheung, Shing-Chi and He, Pinjia and Briand, Lionel}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {149974--150011}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/yu26b/yu26b.pdf}, url = {https://proceedings.mlr.press/v306/yu26b.html}, abstract = {The SWE-Bench Verified leaderboard is approaching saturation, with the top system achieving 78.80%. However, we reveal that this performance is inflated: our re-evaluation demonstrates that one in five “solved” patches from the top-30 agents are semantically incorrect, passing only because weak test suites fail to expose their errors. We present SWE-ABS, an adversarial framework that strengthens test suites through a two-stage pipeline: (1) coverage-driven augmentation utilizing program slicing to target untested code regions, and (2) mutation-driven adversarial testing that synthesizes plausible-but-incorrect patches to expose semantic blind spots. On SWE-Bench Verified (500 instances), SWE-ABS strengthens 50.2% of instances (a $25.1\times$ improvement over prior work) and rejects 19.78% of previously passing patches. Consequently, the top agent’s score decreases from 78.80% to 62.20%, causing significant leaderboard reshuffling (e.g., the top-ranked agent drops to 5th place).} }
Endnote
%0 Conference Paper %T SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark %A Boxi Yu %A Yang Cao %A Yuzhong Zhang %A Liting Lin %A Junjielong Xu %A Zhiqing Zhong %A Qinghua Xu %A Guancheng Wang %A Jialun Cao %A Shing-Chi Cheung %A Pinjia He %A Lionel Briand %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-yu26b %I PMLR %P 149974--150011 %U https://proceedings.mlr.press/v306/yu26b.html %V 306 %X The SWE-Bench Verified leaderboard is approaching saturation, with the top system achieving 78.80%. However, we reveal that this performance is inflated: our re-evaluation demonstrates that one in five “solved” patches from the top-30 agents are semantically incorrect, passing only because weak test suites fail to expose their errors. We present SWE-ABS, an adversarial framework that strengthens test suites through a two-stage pipeline: (1) coverage-driven augmentation utilizing program slicing to target untested code regions, and (2) mutation-driven adversarial testing that synthesizes plausible-but-incorrect patches to expose semantic blind spots. On SWE-Bench Verified (500 instances), SWE-ABS strengthens 50.2% of instances (a $25.1\times$ improvement over prior work) and rejects 19.78% of previously passing patches. Consequently, the top agent’s score decreases from 78.80% to 62.20%, causing significant leaderboard reshuffling (e.g., the top-ranked agent drops to 5th place).
APA
Yu, B., Cao, Y., Zhang, Y., Lin, L., Xu, J., Zhong, Z., Xu, Q., Wang, G., Cao, J., Cheung, S., He, P. & Briand, L.. (2026). SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:149974-150011 Available from https://proceedings.mlr.press/v306/yu26b.html.

Related Material