syftr: Pareto-Optimal Generative AI

Alexander Conway, Debadeepta Dey, Stefan Hackmann, Matthew Hausknecht, Michael Douglas Schmidt, Mark Lewis Steadman, Nick Volynets
Proceedings of the Fourth International Conference on Automated Machine Learning, PMLR 293:6/1-33, 2025.

Abstract

Retrieval-Augmented Generation (RAG) pipelines are central to applying large language models (LLMs) to proprietary or dynamic data. However, building effective RAG flows is complex, requiring careful selection among vector databases, embedding models, text splitters, retrievers, and synthesizing LLMs. The challenge deepens with the rise of agentic paradigms. Modules like verifiers, rewriters, and rerankers—each with intricate hyperparam- eter dependencies have to be carefully tuned. Balancing tradeoffs between latency, accuracy, and cost becomes increasingly difficult in performance-sensitive applications. We introduce syftr, a framework that performs efficient multi-objective search over a broad space of agentic and non-agentic RAG configurations. Using Bayesian Optimization, syftr discovers Pareto-optimal flows that jointly optimize task accuracy and cost. A novel early- stopping mechanism further improves efficiency by pruning clearly suboptimal candidates. Across multiple RAG benchmarks, syftr finds flows which are on average $9\times$ cheaper while preserving most of the accuracy of the most accurate flows on the Pareto-frontier. Furthermore, syftr’s ability to design and optimize also allows integrating new modules, making it even easier and faster to realize high-performing generative AI pipelines. syftr is fully open source: \url{https://github.com/datarobot/syftr}

Cite this Paper

BibTeX
@InProceedings{pmlr-v293-conway25a, title = {syftr: Pareto-Optimal Generative AI}, author = {Conway, Alexander and Dey, Debadeepta and Hackmann, Stefan and Hausknecht, Matthew and Schmidt, Michael Douglas and Steadman, Mark Lewis and Volynets, Nick}, booktitle = {Proceedings of the Fourth International Conference on Automated Machine Learning}, pages = {6/1--33}, year = {2025}, editor = {Akoglu, Leman and Doerr, Carola and van Rijn, Jan N. and Garnett, Roman and Gardner, Jacob R.}, volume = {293}, series = {Proceedings of Machine Learning Research}, month = {08--11 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v293/main/assets/conway25a/conway25a.pdf}, url = {https://proceedings.mlr.press/v293/conway25a.html}, abstract = {Retrieval-Augmented Generation (RAG) pipelines are central to applying large language models (LLMs) to proprietary or dynamic data. However, building effective RAG flows is complex, requiring careful selection among vector databases, embedding models, text splitters, retrievers, and synthesizing LLMs. The challenge deepens with the rise of agentic paradigms. Modules like verifiers, rewriters, and rerankers—each with intricate hyperparam- eter dependencies have to be carefully tuned. Balancing tradeoffs between latency, accuracy, and cost becomes increasingly difficult in performance-sensitive applications. We introduce syftr, a framework that performs efficient multi-objective search over a broad space of agentic and non-agentic RAG configurations. Using Bayesian Optimization, syftr discovers Pareto-optimal flows that jointly optimize task accuracy and cost. A novel early- stopping mechanism further improves efficiency by pruning clearly suboptimal candidates. Across multiple RAG benchmarks, syftr finds flows which are on average $9\times$ cheaper while preserving most of the accuracy of the most accurate flows on the Pareto-frontier. Furthermore, syftr’s ability to design and optimize also allows integrating new modules, making it even easier and faster to realize high-performing generative AI pipelines. syftr is fully open source: \url{https://github.com/datarobot/syftr}} }
Endnote
%0 Conference Paper %T syftr: Pareto-Optimal Generative AI %A Alexander Conway %A Debadeepta Dey %A Stefan Hackmann %A Matthew Hausknecht %A Michael Douglas Schmidt %A Mark Lewis Steadman %A Nick Volynets %B Proceedings of the Fourth International Conference on Automated Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Leman Akoglu %E Carola Doerr %E Jan N. van Rijn %E Roman Garnett %E Jacob R. Gardner %F pmlr-v293-conway25a %I PMLR %P 6/1--33 %U https://proceedings.mlr.press/v293/conway25a.html %V 293 %X Retrieval-Augmented Generation (RAG) pipelines are central to applying large language models (LLMs) to proprietary or dynamic data. However, building effective RAG flows is complex, requiring careful selection among vector databases, embedding models, text splitters, retrievers, and synthesizing LLMs. The challenge deepens with the rise of agentic paradigms. Modules like verifiers, rewriters, and rerankers—each with intricate hyperparam- eter dependencies have to be carefully tuned. Balancing tradeoffs between latency, accuracy, and cost becomes increasingly difficult in performance-sensitive applications. We introduce syftr, a framework that performs efficient multi-objective search over a broad space of agentic and non-agentic RAG configurations. Using Bayesian Optimization, syftr discovers Pareto-optimal flows that jointly optimize task accuracy and cost. A novel early- stopping mechanism further improves efficiency by pruning clearly suboptimal candidates. Across multiple RAG benchmarks, syftr finds flows which are on average $9\times$ cheaper while preserving most of the accuracy of the most accurate flows on the Pareto-frontier. Furthermore, syftr’s ability to design and optimize also allows integrating new modules, making it even easier and faster to realize high-performing generative AI pipelines. syftr is fully open source: \url{https://github.com/datarobot/syftr}
APA
Conway, A., Dey, D., Hackmann, S., Hausknecht, M., Schmidt, M.D., Steadman, M.L. & Volynets, N.. (2025). syftr: Pareto-Optimal Generative AI. Proceedings of the Fourth International Conference on Automated Machine Learning, in Proceedings of Machine Learning Research 293:6/1-33 Available from https://proceedings.mlr.press/v293/conway25a.html.

Related Material