BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics

Dionizije Fa, Marko Čuljak, Bruno Pandža, Mateo Čupić
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:28566-28583, 2026.

Abstract

We introduce BioAgent Bench, an evaluation suite designed for measuring the performance and robustness of AI agents in common bioinformatics tasks. The suite consists of manually curated end-to-end tasks (e.g., RNA-seq, variant calling, metagenomics) accompanied by task-specific prompts and concrete output artifacts to support automated assessment. We evaluate frontier closed- and open-weight models across multiple agent harnesses, and use an LLM-based grader to score pipeline progress and outcome validity. We find that agents based on frontier LLMs can complete multi-step bioinformatics pipelines without elaborate custom scaffolding, often producing the requested final artifacts reliably. However, robustness tests reveal failure modes under controlled perturbations (corrupted inputs, decoy files, and prompt bloat), indicating that correct high-level pipeline construction does not guarantee reliable step-level reasoning. Finally, bioinformatics workflows often involve sensitive patient data or unpublished intellectual property, thereby making the use of cost-effective yet reliable local agents an imperative. Therefore, by releasing the code and the complementary resources comprising our suite, we aim to accelerate the development of such privacy-preserving agents.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-fa26a, title = {{B}io{A}gent Bench: An {AI} Agent Evaluation Suite for Bioinformatics}, author = {Fa, Dionizije and \v{C}uljak, Marko and Pand\v{z}a, Bruno and \v{C}upi\'{c}, Mateo}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {28566--28583}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/fa26a/fa26a.pdf}, url = {https://proceedings.mlr.press/v306/fa26a.html}, abstract = {We introduce BioAgent Bench, an evaluation suite designed for measuring the performance and robustness of AI agents in common bioinformatics tasks. The suite consists of manually curated end-to-end tasks (e.g., RNA-seq, variant calling, metagenomics) accompanied by task-specific prompts and concrete output artifacts to support automated assessment. We evaluate frontier closed- and open-weight models across multiple agent harnesses, and use an LLM-based grader to score pipeline progress and outcome validity. We find that agents based on frontier LLMs can complete multi-step bioinformatics pipelines without elaborate custom scaffolding, often producing the requested final artifacts reliably. However, robustness tests reveal failure modes under controlled perturbations (corrupted inputs, decoy files, and prompt bloat), indicating that correct high-level pipeline construction does not guarantee reliable step-level reasoning. Finally, bioinformatics workflows often involve sensitive patient data or unpublished intellectual property, thereby making the use of cost-effective yet reliable local agents an imperative. Therefore, by releasing the code and the complementary resources comprising our suite, we aim to accelerate the development of such privacy-preserving agents.} }
Endnote
%0 Conference Paper %T BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics %A Dionizije Fa %A Marko Čuljak %A Bruno Pandža %A Mateo Čupić %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-fa26a %I PMLR %P 28566--28583 %U https://proceedings.mlr.press/v306/fa26a.html %V 306 %X We introduce BioAgent Bench, an evaluation suite designed for measuring the performance and robustness of AI agents in common bioinformatics tasks. The suite consists of manually curated end-to-end tasks (e.g., RNA-seq, variant calling, metagenomics) accompanied by task-specific prompts and concrete output artifacts to support automated assessment. We evaluate frontier closed- and open-weight models across multiple agent harnesses, and use an LLM-based grader to score pipeline progress and outcome validity. We find that agents based on frontier LLMs can complete multi-step bioinformatics pipelines without elaborate custom scaffolding, often producing the requested final artifacts reliably. However, robustness tests reveal failure modes under controlled perturbations (corrupted inputs, decoy files, and prompt bloat), indicating that correct high-level pipeline construction does not guarantee reliable step-level reasoning. Finally, bioinformatics workflows often involve sensitive patient data or unpublished intellectual property, thereby making the use of cost-effective yet reliable local agents an imperative. Therefore, by releasing the code and the complementary resources comprising our suite, we aim to accelerate the development of such privacy-preserving agents.
APA
Fa, D., Čuljak, M., Pandža, B. & Čupić, M.. (2026). BioAgent Bench: An AI Agent Evaluation Suite for Bioinformatics. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:28566-28583 Available from https://proceedings.mlr.press/v306/fa26a.html.

Related Material