SWE-IF: Aligning Code Evaluation with Human Preference

Ming Zhong, Xiang Zhou, Ting-Yun Chang, Qingze Wang, Nan Xu, Xiance Si, Dan Garrette, Shyam Upadhyay, Jeremiah Zhe Liu, Jiawei Han, Benoit Schillings, Jiao Sun
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:164657-164692, 2026.

Abstract

Large Language Models (LLMs) have catalyzed vibe coding, where users leverage LLMs to generate and iteratively refine code through natural language interactions until it passes their vibe check. Vibe check reflects human preference and goes beyond functionality: the solution should feel right, read cleanly, preserve intent, and remain correct. However, current code evaluation remains anchored to pass@k and captures only functional correctness, overlooking non-functional instructions that users routinely apply. In this paper, we hypothesize that instruction following is the missing piece underlying vibe check besides functional correctness. To quantify models’ code instruction-following capabilities with measurable signals, we present VeriCode, a taxonomy of 30 verifiable code instructions together with deterministic verifiers. We use the taxonomy to augment established evaluation suites, resulting in SWE-IF, a testbed to assess both instruction following and functional correctness. Evaluating 31 LLMs, we show that even the strongest models struggle to comply with multiple instructions and exhibit functional regression. Most importantly, a composite score of functional correctness and instruction following correlates best with human preference, with instruction following emerging as the primary differentiator among LLMs. Our code, data, and taxonomy are available at https://github.com/maszhongming/SWE-IF.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-zhong26t, title = {{SWE}-{IF}: Aligning Code Evaluation with Human Preference}, author = {Zhong, Ming and Zhou, Xiang and Chang, Ting-Yun and Wang, Qingze and Xu, Nan and Si, Xiance and Garrette, Dan and Upadhyay, Shyam and Liu, Jeremiah Zhe and Han, Jiawei and Schillings, Benoit and Sun, Jiao}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {164657--164692}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/zhong26t/zhong26t.pdf}, url = {https://proceedings.mlr.press/v306/zhong26t.html}, abstract = {Large Language Models (LLMs) have catalyzed vibe coding, where users leverage LLMs to generate and iteratively refine code through natural language interactions until it passes their vibe check. Vibe check reflects human preference and goes beyond functionality: the solution should feel right, read cleanly, preserve intent, and remain correct. However, current code evaluation remains anchored to pass@k and captures only functional correctness, overlooking non-functional instructions that users routinely apply. In this paper, we hypothesize that instruction following is the missing piece underlying vibe check besides functional correctness. To quantify models’ code instruction-following capabilities with measurable signals, we present VeriCode, a taxonomy of 30 verifiable code instructions together with deterministic verifiers. We use the taxonomy to augment established evaluation suites, resulting in SWE-IF, a testbed to assess both instruction following and functional correctness. Evaluating 31 LLMs, we show that even the strongest models struggle to comply with multiple instructions and exhibit functional regression. Most importantly, a composite score of functional correctness and instruction following correlates best with human preference, with instruction following emerging as the primary differentiator among LLMs. Our code, data, and taxonomy are available at https://github.com/maszhongming/SWE-IF.} }
Endnote
%0 Conference Paper %T SWE-IF: Aligning Code Evaluation with Human Preference %A Ming Zhong %A Xiang Zhou %A Ting-Yun Chang %A Qingze Wang %A Nan Xu %A Xiance Si %A Dan Garrette %A Shyam Upadhyay %A Jeremiah Zhe Liu %A Jiawei Han %A Benoit Schillings %A Jiao Sun %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-zhong26t %I PMLR %P 164657--164692 %U https://proceedings.mlr.press/v306/zhong26t.html %V 306 %X Large Language Models (LLMs) have catalyzed vibe coding, where users leverage LLMs to generate and iteratively refine code through natural language interactions until it passes their vibe check. Vibe check reflects human preference and goes beyond functionality: the solution should feel right, read cleanly, preserve intent, and remain correct. However, current code evaluation remains anchored to pass@k and captures only functional correctness, overlooking non-functional instructions that users routinely apply. In this paper, we hypothesize that instruction following is the missing piece underlying vibe check besides functional correctness. To quantify models’ code instruction-following capabilities with measurable signals, we present VeriCode, a taxonomy of 30 verifiable code instructions together with deterministic verifiers. We use the taxonomy to augment established evaluation suites, resulting in SWE-IF, a testbed to assess both instruction following and functional correctness. Evaluating 31 LLMs, we show that even the strongest models struggle to comply with multiple instructions and exhibit functional regression. Most importantly, a composite score of functional correctness and instruction following correlates best with human preference, with instruction following emerging as the primary differentiator among LLMs. Our code, data, and taxonomy are available at https://github.com/maszhongming/SWE-IF.
APA
Zhong, M., Zhou, X., Chang, T., Wang, Q., Xu, N., Si, X., Garrette, D., Upadhyay, S., Liu, J.Z., Han, J., Schillings, B. & Sun, J.. (2026). SWE-IF: Aligning Code Evaluation with Human Preference. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:164657-164692 Available from https://proceedings.mlr.press/v306/zhong26t.html.

Related Material