KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardware

Jiayi Nie, Haoran Wu, Yao Lai, Zeyu Cao, Cheng Zhang, Binglei Lou, Erwei Wang, Jianyi Cheng, Timothy M. Jones, Robert D. Mullins, Rika Antonova, Yiren Zhao
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:93070-93121, 2026.

Abstract

New AI accelerators with novel instruction set architectures (ISAs) often require developers to manually craft low-level kernels — a time-consuming and error-prone process that does not scale across hardware targets. This delays emerging hardware platforms from reaching the market. While prior LLM-based code generation has shown promise in mature GPU ecosystems, it remains unclear whether agentic LLM systems can quickly produce valid and efficient kernels for emerging hardware with new ISAs. We present KernelCraft: the first benchmark for evaluating an LLM agent’s ability to generate and optimize low-level kernels for customized accelerators through a function-calling, feedback-driven workflow. We evaluate agent performance across three emerging accelerators on more than 20 machine-learning tasks, each with five diverse task configurations. Across four leading reasoning models, the strongest agents generate functionally correct kernels for unseen ISAs within a few refinement steps, and produce optimized kernels that match or outperform compiler baselines. These results demonstrate KernelCraft’s potential to accelerate the accelerator chip development cycle. KernelCraft is available at https://kernelcraft-cam.github.io/.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-nie26c, title = {{K}ernel{C}raft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardware}, author = {Nie, Jiayi and Wu, Haoran and Lai, Yao and Cao, Zeyu and Zhang, Cheng and Lou, Binglei and Wang, Erwei and Cheng, Jianyi and Jones, Timothy M. and Mullins, Robert D. and Antonova, Rika and Zhao, Yiren}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {93070--93121}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/nie26c/nie26c.pdf}, url = {https://proceedings.mlr.press/v306/nie26c.html}, abstract = {New AI accelerators with novel instruction set architectures (ISAs) often require developers to manually craft low-level kernels — a time-consuming and error-prone process that does not scale across hardware targets. This delays emerging hardware platforms from reaching the market. While prior LLM-based code generation has shown promise in mature GPU ecosystems, it remains unclear whether agentic LLM systems can quickly produce valid and efficient kernels for emerging hardware with new ISAs. We present KernelCraft: the first benchmark for evaluating an LLM agent’s ability to generate and optimize low-level kernels for customized accelerators through a function-calling, feedback-driven workflow. We evaluate agent performance across three emerging accelerators on more than 20 machine-learning tasks, each with five diverse task configurations. Across four leading reasoning models, the strongest agents generate functionally correct kernels for unseen ISAs within a few refinement steps, and produce optimized kernels that match or outperform compiler baselines. These results demonstrate KernelCraft’s potential to accelerate the accelerator chip development cycle. KernelCraft is available at https://kernelcraft-cam.github.io/.} }
Endnote
%0 Conference Paper %T KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardware %A Jiayi Nie %A Haoran Wu %A Yao Lai %A Zeyu Cao %A Cheng Zhang %A Binglei Lou %A Erwei Wang %A Jianyi Cheng %A Timothy M. Jones %A Robert D. Mullins %A Rika Antonova %A Yiren Zhao %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-nie26c %I PMLR %P 93070--93121 %U https://proceedings.mlr.press/v306/nie26c.html %V 306 %X New AI accelerators with novel instruction set architectures (ISAs) often require developers to manually craft low-level kernels — a time-consuming and error-prone process that does not scale across hardware targets. This delays emerging hardware platforms from reaching the market. While prior LLM-based code generation has shown promise in mature GPU ecosystems, it remains unclear whether agentic LLM systems can quickly produce valid and efficient kernels for emerging hardware with new ISAs. We present KernelCraft: the first benchmark for evaluating an LLM agent’s ability to generate and optimize low-level kernels for customized accelerators through a function-calling, feedback-driven workflow. We evaluate agent performance across three emerging accelerators on more than 20 machine-learning tasks, each with five diverse task configurations. Across four leading reasoning models, the strongest agents generate functionally correct kernels for unseen ISAs within a few refinement steps, and produce optimized kernels that match or outperform compiler baselines. These results demonstrate KernelCraft’s potential to accelerate the accelerator chip development cycle. KernelCraft is available at https://kernelcraft-cam.github.io/.
APA
Nie, J., Wu, H., Lai, Y., Cao, Z., Zhang, C., Lou, B., Wang, E., Cheng, J., Jones, T.M., Mullins, R.D., Antonova, R. & Zhao, Y.. (2026). KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardware. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:93070-93121 Available from https://proceedings.mlr.press/v306/nie26c.html.

Related Material