UniCode: Augmenting Evaluation for Code Reasoning

Xinyue Zheng, Haowei Lin, Shaofei Cai, Yaodong Yang, Zilong Zheng, Yitao Liang
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:163340-163373, 2026.

Abstract

Current coding benchmarks often overstate Large Language Model (LLM) capabilities due to static paradigms and data contamination, allowing models to exploit statistical shortcuts rather than genuine reasoning. To address this, we introduce UniCode, a generative evaluation framework that systematically probes LLM reasoning boundaries via: (1) multi-dimensional augmentation operators to create diverse algorithmic variants; (2) a scalable test generation pipeline achieving 94.5% correctness without human-written solutions; and (3) fine-grained diagnostic metrics for rich error signals. Our evaluation of state-of-the-art models reveals a significant 31.2% performance collapse. Critically, we observe a high variance across different reasoning axes, revealing a profound fragility under structural shifts despite surface-level robustness. Furthermore, we identify a “seed-problem regression," where models fail by defaulting to memorized seed logic and inefficient complexities. Our evaluation code is publicly available at https://github.com/grandsmile/UniCode.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-zheng26i, title = {{U}ni{C}ode: Augmenting Evaluation for Code Reasoning}, author = {Zheng, Xinyue and Lin, Haowei and Cai, Shaofei and Yang, Yaodong and Zheng, Zilong and Liang, Yitao}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {163340--163373}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/zheng26i/zheng26i.pdf}, url = {https://proceedings.mlr.press/v306/zheng26i.html}, abstract = {Current coding benchmarks often overstate Large Language Model (LLM) capabilities due to static paradigms and data contamination, allowing models to exploit statistical shortcuts rather than genuine reasoning. To address this, we introduce UniCode, a generative evaluation framework that systematically probes LLM reasoning boundaries via: (1) multi-dimensional augmentation operators to create diverse algorithmic variants; (2) a scalable test generation pipeline achieving 94.5% correctness without human-written solutions; and (3) fine-grained diagnostic metrics for rich error signals. Our evaluation of state-of-the-art models reveals a significant 31.2% performance collapse. Critically, we observe a high variance across different reasoning axes, revealing a profound fragility under structural shifts despite surface-level robustness. Furthermore, we identify a “seed-problem regression," where models fail by defaulting to memorized seed logic and inefficient complexities. Our evaluation code is publicly available at https://github.com/grandsmile/UniCode.} }
Endnote
%0 Conference Paper %T UniCode: Augmenting Evaluation for Code Reasoning %A Xinyue Zheng %A Haowei Lin %A Shaofei Cai %A Yaodong Yang %A Zilong Zheng %A Yitao Liang %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-zheng26i %I PMLR %P 163340--163373 %U https://proceedings.mlr.press/v306/zheng26i.html %V 306 %X Current coding benchmarks often overstate Large Language Model (LLM) capabilities due to static paradigms and data contamination, allowing models to exploit statistical shortcuts rather than genuine reasoning. To address this, we introduce UniCode, a generative evaluation framework that systematically probes LLM reasoning boundaries via: (1) multi-dimensional augmentation operators to create diverse algorithmic variants; (2) a scalable test generation pipeline achieving 94.5% correctness without human-written solutions; and (3) fine-grained diagnostic metrics for rich error signals. Our evaluation of state-of-the-art models reveals a significant 31.2% performance collapse. Critically, we observe a high variance across different reasoning axes, revealing a profound fragility under structural shifts despite surface-level robustness. Furthermore, we identify a “seed-problem regression," where models fail by defaulting to memorized seed logic and inefficient complexities. Our evaluation code is publicly available at https://github.com/grandsmile/UniCode.
APA
Zheng, X., Lin, H., Cai, S., Yang, Y., Zheng, Z. & Liang, Y.. (2026). UniCode: Augmenting Evaluation for Code Reasoning. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:163340-163373 Available from https://proceedings.mlr.press/v306/zheng26i.html.

Related Material