BEST: Benchmarking Efficiency in Space and Time for LLM-Generated Code

Aocheng Shen, Boyu Zhang, Jiaze Li, Ruixuan Ma, Qiankun Zhang, Jing Wang, Bin Yuan, Shenghao Liu, Xianjun Deng
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:111476-111509, 2026.

Abstract

Large language models (LLMs) have revolutionized research in software engineering, and among various tasks, LLM-based code synthesis is promising. A recent line of benchmarks aims to evaluate LLM-generated codes in time efficiency, beyond their correctness. However, space, another vital aspect of code efficiency, is rarely evaluated in prior benchmarks. To fill in the gap, this paper introduces BEST, the first benchmark for evaluating the efficiency of LLM-generated codes in both time and space. It comprises $440$ coding tasks that are rigorously constructed by experts. In addition, we propose a fine-grained subtask-based evaluation scheme by dividing each task into multiple subtasks, with different input scales and difficulties. Each subtask is then accompanied by an expert-crafted standard implementation as the efficiency baseline, which achieves the Pareto optimum. Building on BEST, we introduce a unified and novel dual-indicator (time and space) metric, named dual@${k}$, generalizing the notion of the standard pass@${k}$ metric and building on a careful and novel construction of a weight matrix of subtasks. Through extensive experiments with dual@${k}$ across $50$ LLMs on BEST, our evaluation demonstrates that while LLMs exhibit weak capabilities in generating time-efficient code, their capabilities in space-efficient code generation are even worse. The benchmark is provided at https://github.com/kmsgk0/BEST.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-shen26ac, title = {{BEST}: Benchmarking Efficiency in Space and Time for {LLM}-Generated Code}, author = {Shen, Aocheng and Zhang, Boyu and Li, Jiaze and Ma, Ruixuan and Zhang, Qiankun and Wang, Jing and Yuan, Bin and Liu, Shenghao and Deng, Xianjun}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {111476--111509}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/shen26ac/shen26ac.pdf}, url = {https://proceedings.mlr.press/v306/shen26ac.html}, abstract = {Large language models (LLMs) have revolutionized research in software engineering, and among various tasks, LLM-based code synthesis is promising. A recent line of benchmarks aims to evaluate LLM-generated codes in time efficiency, beyond their correctness. However, space, another vital aspect of code efficiency, is rarely evaluated in prior benchmarks. To fill in the gap, this paper introduces BEST, the first benchmark for evaluating the efficiency of LLM-generated codes in both time and space. It comprises $440$ coding tasks that are rigorously constructed by experts. In addition, we propose a fine-grained subtask-based evaluation scheme by dividing each task into multiple subtasks, with different input scales and difficulties. Each subtask is then accompanied by an expert-crafted standard implementation as the efficiency baseline, which achieves the Pareto optimum. Building on BEST, we introduce a unified and novel dual-indicator (time and space) metric, named dual@${k}$, generalizing the notion of the standard pass@${k}$ metric and building on a careful and novel construction of a weight matrix of subtasks. Through extensive experiments with dual@${k}$ across $50$ LLMs on BEST, our evaluation demonstrates that while LLMs exhibit weak capabilities in generating time-efficient code, their capabilities in space-efficient code generation are even worse. The benchmark is provided at https://github.com/kmsgk0/BEST.} }
Endnote
%0 Conference Paper %T BEST: Benchmarking Efficiency in Space and Time for LLM-Generated Code %A Aocheng Shen %A Boyu Zhang %A Jiaze Li %A Ruixuan Ma %A Qiankun Zhang %A Jing Wang %A Bin Yuan %A Shenghao Liu %A Xianjun Deng %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-shen26ac %I PMLR %P 111476--111509 %U https://proceedings.mlr.press/v306/shen26ac.html %V 306 %X Large language models (LLMs) have revolutionized research in software engineering, and among various tasks, LLM-based code synthesis is promising. A recent line of benchmarks aims to evaluate LLM-generated codes in time efficiency, beyond their correctness. However, space, another vital aspect of code efficiency, is rarely evaluated in prior benchmarks. To fill in the gap, this paper introduces BEST, the first benchmark for evaluating the efficiency of LLM-generated codes in both time and space. It comprises $440$ coding tasks that are rigorously constructed by experts. In addition, we propose a fine-grained subtask-based evaluation scheme by dividing each task into multiple subtasks, with different input scales and difficulties. Each subtask is then accompanied by an expert-crafted standard implementation as the efficiency baseline, which achieves the Pareto optimum. Building on BEST, we introduce a unified and novel dual-indicator (time and space) metric, named dual@${k}$, generalizing the notion of the standard pass@${k}$ metric and building on a careful and novel construction of a weight matrix of subtasks. Through extensive experiments with dual@${k}$ across $50$ LLMs on BEST, our evaluation demonstrates that while LLMs exhibit weak capabilities in generating time-efficient code, their capabilities in space-efficient code generation are even worse. The benchmark is provided at https://github.com/kmsgk0/BEST.
APA
Shen, A., Zhang, B., Li, J., Ma, R., Zhang, Q., Wang, J., Yuan, B., Liu, S. & Deng, X.. (2026). BEST: Benchmarking Efficiency in Space and Time for LLM-Generated Code. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:111476-111509 Available from https://proceedings.mlr.press/v306/shen26ac.html.

Related Material