FormulaCode: Evaluating Agentic Optimization on Large Codebases

Atharva Sehgal, James Hou, Akanksha Sarkar, Ishaan Mantripragada, Swarat Chaudhuri, Jennifer J. Sun, Yisong Yue
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:108728-108783, 2026.

Abstract

Large language model (LLM) coding agents increasingly operate at the repository level, motivating benchmarks that evaluate their ability to optimize entire codebases under realistic constraints. Existing code benchmarks largely rely on synthetic tasks, binary correctness signals, or single-objective evaluation, limiting their ability to assess holistic optimization behavior. We introduce FormulaCode, a benchmark for evaluating agentic optimization on large, real-world codebases with fine-grained, multi-objective performance metrics. FormulaCode comprises 957 performance bottlenecks mined from scientific Python repositories on GitHub, each paired with expert-authored patches and 264.6 community-maintained performance workloads per task, enabling evaluation of the full optimization lifecycle—triage, diagnosis, and resolution—under realistic correctness and performance constraints. Our evaluations reveal that repository-scale, multi-objective optimization remains a major challenge for frontier LLM agents.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-sehgal26a, title = {{F}ormula{C}ode: Evaluating Agentic Optimization on Large Codebases}, author = {Sehgal, Atharva and Hou, James and Sarkar, Akanksha and Mantripragada, Ishaan and Chaudhuri, Swarat and Sun, Jennifer J. and Yue, Yisong}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {108728--108783}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/sehgal26a/sehgal26a.pdf}, url = {https://proceedings.mlr.press/v306/sehgal26a.html}, abstract = {Large language model (LLM) coding agents increasingly operate at the repository level, motivating benchmarks that evaluate their ability to optimize entire codebases under realistic constraints. Existing code benchmarks largely rely on synthetic tasks, binary correctness signals, or single-objective evaluation, limiting their ability to assess holistic optimization behavior. We introduce FormulaCode, a benchmark for evaluating agentic optimization on large, real-world codebases with fine-grained, multi-objective performance metrics. FormulaCode comprises 957 performance bottlenecks mined from scientific Python repositories on GitHub, each paired with expert-authored patches and 264.6 community-maintained performance workloads per task, enabling evaluation of the full optimization lifecycle—triage, diagnosis, and resolution—under realistic correctness and performance constraints. Our evaluations reveal that repository-scale, multi-objective optimization remains a major challenge for frontier LLM agents.} }
Endnote
%0 Conference Paper %T FormulaCode: Evaluating Agentic Optimization on Large Codebases %A Atharva Sehgal %A James Hou %A Akanksha Sarkar %A Ishaan Mantripragada %A Swarat Chaudhuri %A Jennifer J. Sun %A Yisong Yue %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-sehgal26a %I PMLR %P 108728--108783 %U https://proceedings.mlr.press/v306/sehgal26a.html %V 306 %X Large language model (LLM) coding agents increasingly operate at the repository level, motivating benchmarks that evaluate their ability to optimize entire codebases under realistic constraints. Existing code benchmarks largely rely on synthetic tasks, binary correctness signals, or single-objective evaluation, limiting their ability to assess holistic optimization behavior. We introduce FormulaCode, a benchmark for evaluating agentic optimization on large, real-world codebases with fine-grained, multi-objective performance metrics. FormulaCode comprises 957 performance bottlenecks mined from scientific Python repositories on GitHub, each paired with expert-authored patches and 264.6 community-maintained performance workloads per task, enabling evaluation of the full optimization lifecycle—triage, diagnosis, and resolution—under realistic correctness and performance constraints. Our evaluations reveal that repository-scale, multi-objective optimization remains a major challenge for frontier LLM agents.
APA
Sehgal, A., Hou, J., Sarkar, A., Mantripragada, I., Chaudhuri, S., Sun, J.J. & Yue, Y.. (2026). FormulaCode: Evaluating Agentic Optimization on Large Codebases. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:108728-108783 Available from https://proceedings.mlr.press/v306/sehgal26a.html.

Related Material