Procedural Generation Of Algorithm Discovery Tasks in Machine Learning

Alexander David Goldie, Zilin Wang, Adrian Hayler, Deepak Nathani, Edan Toledo, Ken Thampiratwong, Aleksandra Kalisz, Michael Beukman, Alistair Letcher, Shashank Reddy Chirra, Clarisse Wibault, Theo Wolf, Charles O’Neill, Uljad Berdica, Nicholas Roberts, Saeed Rahmani, Roberta Raileanu, Shimon Whiteson, Jakob Nicolaus Foerster
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:35600-35715, 2026.

Abstract

Automating the development of machine learning algorithms has the potential to unlock new breakthroughs. However, our ability to improve and evaluate algorithm discovery systems has thus far been limited by existing task suites. They suffer from many issues, such as: poor evaluation methodologies; data contamination; and containing saturated or very similar problems. Here, we introduce DiscoGen, a procedural generator of algorithm discovery tasks for machine learning, such as developing optimisers for reinforcement learning or loss functions for image classification. Motivated by the success of procedural generation in reinforcement learning, DiscoGen spans billions of tasks of varying difficulty and complexity from a range of machine learning fields. These tasks are specified by a small number of configuration parameters and can be used to optimise algorithm discovery agents (ADAs). We present DiscoBench, a fixed, small subset of DiscoGen tasks for principled evaluation of ADAs. Finally, we propose a number of ambitious, impactful research directions enabled by DiscoGen, and demonstrate its use for ADA optimisation through scaling experiments for automated prompt tuning. DiscoGen is released open-source.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-goldie26a, title = {Procedural Generation Of Algorithm Discovery Tasks in Machine Learning}, author = {Goldie, Alexander David and Wang, Zilin and Hayler, Adrian and Nathani, Deepak and Toledo, Edan and Thampiratwong, Ken and Kalisz, Aleksandra and Beukman, Michael and Letcher, Alistair and Reddy Chirra, Shashank and Wibault, Clarisse and Wolf, Theo and O'Neill, Charles and Berdica, Uljad and Roberts, Nicholas and Rahmani, Saeed and Raileanu, Roberta and Whiteson, Shimon and Foerster, Jakob Nicolaus}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {35600--35715}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/goldie26a/goldie26a.pdf}, url = {https://proceedings.mlr.press/v306/goldie26a.html}, abstract = {Automating the development of machine learning algorithms has the potential to unlock new breakthroughs. However, our ability to improve and evaluate algorithm discovery systems has thus far been limited by existing task suites. They suffer from many issues, such as: poor evaluation methodologies; data contamination; and containing saturated or very similar problems. Here, we introduce DiscoGen, a procedural generator of algorithm discovery tasks for machine learning, such as developing optimisers for reinforcement learning or loss functions for image classification. Motivated by the success of procedural generation in reinforcement learning, DiscoGen spans billions of tasks of varying difficulty and complexity from a range of machine learning fields. These tasks are specified by a small number of configuration parameters and can be used to optimise algorithm discovery agents (ADAs). We present DiscoBench, a fixed, small subset of DiscoGen tasks for principled evaluation of ADAs. Finally, we propose a number of ambitious, impactful research directions enabled by DiscoGen, and demonstrate its use for ADA optimisation through scaling experiments for automated prompt tuning. DiscoGen is released open-source.} }
Endnote
%0 Conference Paper %T Procedural Generation Of Algorithm Discovery Tasks in Machine Learning %A Alexander David Goldie %A Zilin Wang %A Adrian Hayler %A Deepak Nathani %A Edan Toledo %A Ken Thampiratwong %A Aleksandra Kalisz %A Michael Beukman %A Alistair Letcher %A Shashank Reddy Chirra %A Clarisse Wibault %A Theo Wolf %A Charles O’Neill %A Uljad Berdica %A Nicholas Roberts %A Saeed Rahmani %A Roberta Raileanu %A Shimon Whiteson %A Jakob Nicolaus Foerster %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-goldie26a %I PMLR %P 35600--35715 %U https://proceedings.mlr.press/v306/goldie26a.html %V 306 %X Automating the development of machine learning algorithms has the potential to unlock new breakthroughs. However, our ability to improve and evaluate algorithm discovery systems has thus far been limited by existing task suites. They suffer from many issues, such as: poor evaluation methodologies; data contamination; and containing saturated or very similar problems. Here, we introduce DiscoGen, a procedural generator of algorithm discovery tasks for machine learning, such as developing optimisers for reinforcement learning or loss functions for image classification. Motivated by the success of procedural generation in reinforcement learning, DiscoGen spans billions of tasks of varying difficulty and complexity from a range of machine learning fields. These tasks are specified by a small number of configuration parameters and can be used to optimise algorithm discovery agents (ADAs). We present DiscoBench, a fixed, small subset of DiscoGen tasks for principled evaluation of ADAs. Finally, we propose a number of ambitious, impactful research directions enabled by DiscoGen, and demonstrate its use for ADA optimisation through scaling experiments for automated prompt tuning. DiscoGen is released open-source.
APA
Goldie, A.D., Wang, Z., Hayler, A., Nathani, D., Toledo, E., Thampiratwong, K., Kalisz, A., Beukman, M., Letcher, A., Reddy Chirra, S., Wibault, C., Wolf, T., O’Neill, C., Berdica, U., Roberts, N., Rahmani, S., Raileanu, R., Whiteson, S. & Foerster, J.N.. (2026). Procedural Generation Of Algorithm Discovery Tasks in Machine Learning. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:35600-35715 Available from https://proceedings.mlr.press/v306/goldie26a.html.

Related Material