Scaling Laws and Architectural Frontiers in Metagenomic Foundation Models

Geraldene Munsamy, Gavin Ayres, Jérémie Dona, Carla Greco, Daniel Anderson, Srijani Sridhar, William Chow, Aaron W Kollasch, Robert J. Pecoraro, Tanggis Bohnuud, Keith Kam, Gus Minto-Cowcher, Marcus H Y Leung, Hassan Sirelkhatim, John St. John, Ali Taghibakhshi, Tyler C. Shimko, Jared Wilber, Timur Rvachov, Saee Gopal Paliwal, Eduardo Calleja, Noelia Ferruz, Kevin K Yang, Philipp Lorenz, Francesco Farina
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:91276-91292, 2026.

Abstract

Foundation models for genomics have the potential to revolutionize therapeutic design, yet the optimal architectural choices for modeling the vast and diverse distribution of metagenomic data remain under-explored. In this work, we present the machine learning methodology behind EDEN, a family of metagenomic foundation models scaled up to 28 billion parameters and trained on 9.7 trillion nucleotide tokens. We provide a systematic empirical study of architectural trade-offs between autoregressive Transformers (Llama-style), State-Space Models (Mamba), and Long-convolutional architectures (Hyena) for nucleotide-level modeling. Contrary to recent trends favoring linear-time sequence models for long-range biological data, we demonstrate that the Llama architecture exhibits superior scaling efficiency and semantic retrieval capabilities as the model capacity grows. We derive a set of quality-aware scaling laws for metagenomics, showing how model performance follows predictable power-law behavior across three orders of magnitude in parameters and data. Through extensive benchmarking, spanning unsupervised zeroshot fitness prediction, semantic completion, and gene recovery, we establish a blueprint for scaling biological foundation models and provide empirical evidence demonstrating why Transformer-based architectures define the current frontier.

Cite this Paper


BibTeX
@InProceedings{pmlr-v306-munsamy26a, title = {Scaling Laws and Architectural Frontiers in Metagenomic Foundation Models}, author = {Munsamy, Geraldene and Ayres, Gavin and Dona, J\'{e}r\'{e}mie and Greco, Carla and Anderson, Daniel and Sridhar, Srijani and Chow, William and Kollasch, Aaron W and Pecoraro, Robert J. and Bohnuud, Tanggis and Kam, Keith and Minto-Cowcher, Gus and Leung, Marcus H Y and Sirelkhatim, Hassan and John, John St. and Taghibakhshi, Ali and Shimko, Tyler C. and Wilber, Jared and Rvachov, Timur and Paliwal, Saee Gopal and Calleja, Eduardo and Ferruz, Noelia and Yang, Kevin K and Lorenz, Philipp and Farina, Francesco}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {91276--91292}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/munsamy26a/munsamy26a.pdf}, url = {https://proceedings.mlr.press/v306/munsamy26a.html}, abstract = {Foundation models for genomics have the potential to revolutionize therapeutic design, yet the optimal architectural choices for modeling the vast and diverse distribution of metagenomic data remain under-explored. In this work, we present the machine learning methodology behind EDEN, a family of metagenomic foundation models scaled up to 28 billion parameters and trained on 9.7 trillion nucleotide tokens. We provide a systematic empirical study of architectural trade-offs between autoregressive Transformers (Llama-style), State-Space Models (Mamba), and Long-convolutional architectures (Hyena) for nucleotide-level modeling. Contrary to recent trends favoring linear-time sequence models for long-range biological data, we demonstrate that the Llama architecture exhibits superior scaling efficiency and semantic retrieval capabilities as the model capacity grows. We derive a set of quality-aware scaling laws for metagenomics, showing how model performance follows predictable power-law behavior across three orders of magnitude in parameters and data. Through extensive benchmarking, spanning unsupervised zeroshot fitness prediction, semantic completion, and gene recovery, we establish a blueprint for scaling biological foundation models and provide empirical evidence demonstrating why Transformer-based architectures define the current frontier.} }
Endnote
%0 Conference Paper %T Scaling Laws and Architectural Frontiers in Metagenomic Foundation Models %A Geraldene Munsamy %A Gavin Ayres %A Jérémie Dona %A Carla Greco %A Daniel Anderson %A Srijani Sridhar %A William Chow %A Aaron W Kollasch %A Robert J. Pecoraro %A Tanggis Bohnuud %A Keith Kam %A Gus Minto-Cowcher %A Marcus H Y Leung %A Hassan Sirelkhatim %A John St. John %A Ali Taghibakhshi %A Tyler C. Shimko %A Jared Wilber %A Timur Rvachov %A Saee Gopal Paliwal %A Eduardo Calleja %A Noelia Ferruz %A Kevin K Yang %A Philipp Lorenz %A Francesco Farina %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-munsamy26a %I PMLR %P 91276--91292 %U https://proceedings.mlr.press/v306/munsamy26a.html %V 306 %X Foundation models for genomics have the potential to revolutionize therapeutic design, yet the optimal architectural choices for modeling the vast and diverse distribution of metagenomic data remain under-explored. In this work, we present the machine learning methodology behind EDEN, a family of metagenomic foundation models scaled up to 28 billion parameters and trained on 9.7 trillion nucleotide tokens. We provide a systematic empirical study of architectural trade-offs between autoregressive Transformers (Llama-style), State-Space Models (Mamba), and Long-convolutional architectures (Hyena) for nucleotide-level modeling. Contrary to recent trends favoring linear-time sequence models for long-range biological data, we demonstrate that the Llama architecture exhibits superior scaling efficiency and semantic retrieval capabilities as the model capacity grows. We derive a set of quality-aware scaling laws for metagenomics, showing how model performance follows predictable power-law behavior across three orders of magnitude in parameters and data. Through extensive benchmarking, spanning unsupervised zeroshot fitness prediction, semantic completion, and gene recovery, we establish a blueprint for scaling biological foundation models and provide empirical evidence demonstrating why Transformer-based architectures define the current frontier.
APA
Munsamy, G., Ayres, G., Dona, J., Greco, C., Anderson, D., Sridhar, S., Chow, W., Kollasch, A.W., Pecoraro, R.J., Bohnuud, T., Kam, K., Minto-Cowcher, G., Leung, M.H.Y., Sirelkhatim, H., John, J.S., Taghibakhshi, A., Shimko, T.C., Wilber, J., Rvachov, T., Paliwal, S.G., Calleja, E., Ferruz, N., Yang, K.K., Lorenz, P. & Farina, F.. (2026). Scaling Laws and Architectural Frontiers in Metagenomic Foundation Models. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:91276-91292 Available from https://proceedings.mlr.press/v306/munsamy26a.html.

Related Material