FrontierCS: Evolving Challenges for Evolving Intelligence

Qiuyang Mang, Wenhao Chai, Zhifei Li, Huanzhi Mao, Shang Zhou, Alexander Du, Hanchen Li, Shu Liu, Edwin Chen, Yichuan Wang, Xieting Chu, Zerui Cheng, Yuan Xu, Tian Xia, Zirui Wang, Tianneng Shi, Jianzhu Yao, Yilong Zhao, Qizheng Zhang, Charlie F. Ruan, Zeyu Shen, Kaiyuan Liu, Zhaoyang Hong, Alex Gu, Ziyi Zhang, Runyuan He, Dong Xing, Zerui Li, Zirong Zeng, Yige Jiang, Lufeng Cheng, Ziyi Zhao, Youran Sun, Suyang Zhong, Junpeng Wang, Donglin Li, Wenyuan Huang, Jialiang Gu, Wesley Kai Zheng, Wangmeiyu Zhang, Ruyi Ji, Xuechang Tu, Zihan Zheng, Zhaozi Wang, Zexing Chen, Jingbang Chen, Jialu Zhang, Aleksandra Korolova, Peter Henderson, Pramod Viswanath, Vijay Ganesh, Saining Xie, Zhuang Liu, Dawn Song, Sewon Min, Ion Stoica, Joseph E. Gonzalez, Jingbo Shang, Alvin Cheung
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:86255-86277, 2026.

Abstract

We introduce FrontierCS, a benchmark of 240 open-ended problems across diverse areas of computer science, designed and reviewed by experts, including CS PhDs and top-tier competitive programming participants and problem setters. Unlike existing benchmarks that focus on tasks with known optimal solutions, FrontierCS targets problems where the optimal solution is unknown, but the quality of a solution can be objectively evaluated. Models solve these tasks by implementing executable programs rather than outputting a direct answer. FrontierCS includes algorithmic problems, which are often NP-hard variants of competitive programming problems with objective partial scoring, and research problems with the same property. For each problem, we provide an expert reference solution and an automatic evaluator. Combining open-ended design, measurable progress, and expert curation, FrontierCS provides a benchmark at the frontier of computer-science difficulty. Empirically, we find that frontier reasoning models still lag far behind human experts, and that simply increasing reasoning budgets does not close this gap on open-ended challenges. Moreover, these models struggle to identify internal equivalence classes, and existing agentic frameworks also exhibit brittleness on such problems due to overfitting. FrontierCS thus offers a new lens into model capabilities on real frontier computer science problems.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-mang26a, title = {{F}rontier{CS}: Evolving Challenges for Evolving Intelligence}, author = {Mang, Qiuyang and Chai, Wenhao and Li, Zhifei and Mao, Huanzhi and Zhou, Shang and Du, Alexander and Li, Hanchen and Liu, Shu and Chen, Edwin and Wang, Yichuan and Chu, Xieting and Cheng, Zerui and Xu, Yuan and Xia, Tian and Wang, Zirui and Shi, Tianneng and Yao, Jianzhu and Zhao, Yilong and Zhang, Qizheng and Ruan, Charlie F. and Shen, Zeyu and Liu, Kaiyuan and Hong, Zhaoyang and Gu, Alex and Zhang, Ziyi and He, Runyuan and Xing, Dong and Li, Zerui and Zeng, Zirong and Jiang, Yige and Cheng, Lufeng and Zhao, Ziyi and Sun, Youran and Zhong, Suyang and Wang, Junpeng and Li, Donglin and Huang, Wenyuan and Gu, Jialiang and Zheng, Wesley Kai and Zhang, Wangmeiyu and Ji, Ruyi and Tu, Xuechang and Zheng, Zihan and Wang, Zhaozi and Chen, Zexing and Chen, Jingbang and Zhang, Jialu and Korolova, Aleksandra and Henderson, Peter and Viswanath, Pramod and Ganesh, Vijay and Xie, Saining and Liu, Zhuang and Song, Dawn and Min, Sewon and Stoica, Ion and Gonzalez, Joseph E. and Shang, Jingbo and Cheung, Alvin}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {86255--86277}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/mang26a/mang26a.pdf}, url = {https://proceedings.mlr.press/v306/mang26a.html}, abstract = {We introduce FrontierCS, a benchmark of 240 open-ended problems across diverse areas of computer science, designed and reviewed by experts, including CS PhDs and top-tier competitive programming participants and problem setters. Unlike existing benchmarks that focus on tasks with known optimal solutions, FrontierCS targets problems where the optimal solution is unknown, but the quality of a solution can be objectively evaluated. Models solve these tasks by implementing executable programs rather than outputting a direct answer. FrontierCS includes algorithmic problems, which are often NP-hard variants of competitive programming problems with objective partial scoring, and research problems with the same property. For each problem, we provide an expert reference solution and an automatic evaluator. Combining open-ended design, measurable progress, and expert curation, FrontierCS provides a benchmark at the frontier of computer-science difficulty. Empirically, we find that frontier reasoning models still lag far behind human experts, and that simply increasing reasoning budgets does not close this gap on open-ended challenges. Moreover, these models struggle to identify internal equivalence classes, and existing agentic frameworks also exhibit brittleness on such problems due to overfitting. FrontierCS thus offers a new lens into model capabilities on real frontier computer science problems.} }
Endnote
%0 Conference Paper %T FrontierCS: Evolving Challenges for Evolving Intelligence %A Qiuyang Mang %A Wenhao Chai %A Zhifei Li %A Huanzhi Mao %A Shang Zhou %A Alexander Du %A Hanchen Li %A Shu Liu %A Edwin Chen %A Yichuan Wang %A Xieting Chu %A Zerui Cheng %A Yuan Xu %A Tian Xia %A Zirui Wang %A Tianneng Shi %A Jianzhu Yao %A Yilong Zhao %A Qizheng Zhang %A Charlie F. Ruan %A Zeyu Shen %A Kaiyuan Liu %A Zhaoyang Hong %A Alex Gu %A Ziyi Zhang %A Runyuan He %A Dong Xing %A Zerui Li %A Zirong Zeng %A Yige Jiang %A Lufeng Cheng %A Ziyi Zhao %A Youran Sun %A Suyang Zhong %A Junpeng Wang %A Donglin Li %A Wenyuan Huang %A Jialiang Gu %A Wesley Kai Zheng %A Wangmeiyu Zhang %A Ruyi Ji %A Xuechang Tu %A Zihan Zheng %A Zhaozi Wang %A Zexing Chen %A Jingbang Chen %A Jialu Zhang %A Aleksandra Korolova %A Peter Henderson %A Pramod Viswanath %A Vijay Ganesh %A Saining Xie %A Zhuang Liu %A Dawn Song %A Sewon Min %A Ion Stoica %A Joseph E. Gonzalez %A Jingbo Shang %A Alvin Cheung %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-mang26a %I PMLR %P 86255--86277 %U https://proceedings.mlr.press/v306/mang26a.html %V 306 %X We introduce FrontierCS, a benchmark of 240 open-ended problems across diverse areas of computer science, designed and reviewed by experts, including CS PhDs and top-tier competitive programming participants and problem setters. Unlike existing benchmarks that focus on tasks with known optimal solutions, FrontierCS targets problems where the optimal solution is unknown, but the quality of a solution can be objectively evaluated. Models solve these tasks by implementing executable programs rather than outputting a direct answer. FrontierCS includes algorithmic problems, which are often NP-hard variants of competitive programming problems with objective partial scoring, and research problems with the same property. For each problem, we provide an expert reference solution and an automatic evaluator. Combining open-ended design, measurable progress, and expert curation, FrontierCS provides a benchmark at the frontier of computer-science difficulty. Empirically, we find that frontier reasoning models still lag far behind human experts, and that simply increasing reasoning budgets does not close this gap on open-ended challenges. Moreover, these models struggle to identify internal equivalence classes, and existing agentic frameworks also exhibit brittleness on such problems due to overfitting. FrontierCS thus offers a new lens into model capabilities on real frontier computer science problems.
APA
Mang, Q., Chai, W., Li, Z., Mao, H., Zhou, S., Du, A., Li, H., Liu, S., Chen, E., Wang, Y., Chu, X., Cheng, Z., Xu, Y., Xia, T., Wang, Z., Shi, T., Yao, J., Zhao, Y., Zhang, Q., Ruan, C.F., Shen, Z., Liu, K., Hong, Z., Gu, A., Zhang, Z., He, R., Xing, D., Li, Z., Zeng, Z., Jiang, Y., Cheng, L., Zhao, Z., Sun, Y., Zhong, S., Wang, J., Li, D., Huang, W., Gu, J., Zheng, W.K., Zhang, W., Ji, R., Tu, X., Zheng, Z., Wang, Z., Chen, Z., Chen, J., Zhang, J., Korolova, A., Henderson, P., Viswanath, P., Ganesh, V., Xie, S., Liu, Z., Song, D., Min, S., Stoica, I., Gonzalez, J.E., Shang, J. & Cheung, A.. (2026). FrontierCS: Evolving Challenges for Evolving Intelligence. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:86255-86277 Available from https://proceedings.mlr.press/v306/mang26a.html.

Related Material