AutoPDL: Automatic Prompt Optimization for LLM Agents

Claudio Spiess, Mandana Vaziri, Louis Mandel, Martin Hirzel
Proceedings of the Fourth International Conference on Automated Machine Learning, PMLR 293:13/1-20, 2025.

Abstract

The performance of large language models (LLMs) depends on how they are prompted, with choices spanning both the high-level prompting pattern (e.g., Zero-Shot, CoT, ReAct, ReWOO) and the specific prompt content (instructions and few-shot demonstrations). Manually tuning this combination is tedious, error-prone, and non-transferable across LLMs or tasks. Therefore, this paper proposes AutoPDL, an automated approach to discover good LLM agent configurations. Our method frames this as a structured AutoML problem over a combinatorial space of agentic and non-agentic prompting patterns and demonstrations, using successive halving to efficiently navigate this space. We introduce a library implementing common prompting patterns using the PDL prompt programming language. AutoPDL solutions are human-readable, editable, and executable PDL programs that use this library. This approach also enables source-to-source optimization, allowing human-in-the-loop refinement and reuse. Evaluations across three tasks and six LLMs (ranging from 3B to 70B parameters) show consistent accuracy gains ($9.06 \pm 15.3$ percentage points), up to 68.9pp, and reveal that selected prompting strategies vary across models and tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v293-spiess25a, title = {AutoPDL: Automatic Prompt Optimization for LLM Agents}, author = {Spiess, Claudio and Vaziri, Mandana and Mandel, Louis and Hirzel, Martin}, booktitle = {Proceedings of the Fourth International Conference on Automated Machine Learning}, pages = {13/1--20}, year = {2025}, editor = {Akoglu, Leman and Doerr, Carola and van Rijn, Jan N. and Garnett, Roman and Gardner, Jacob R.}, volume = {293}, series = {Proceedings of Machine Learning Research}, month = {08--11 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v293/main/assets/spiess25a/spiess25a.pdf}, url = {https://proceedings.mlr.press/v293/spiess25a.html}, abstract = {The performance of large language models (LLMs) depends on how they are prompted, with choices spanning both the high-level prompting pattern (e.g., Zero-Shot, CoT, ReAct, ReWOO) and the specific prompt content (instructions and few-shot demonstrations). Manually tuning this combination is tedious, error-prone, and non-transferable across LLMs or tasks. Therefore, this paper proposes AutoPDL, an automated approach to discover good LLM agent configurations. Our method frames this as a structured AutoML problem over a combinatorial space of agentic and non-agentic prompting patterns and demonstrations, using successive halving to efficiently navigate this space. We introduce a library implementing common prompting patterns using the PDL prompt programming language. AutoPDL solutions are human-readable, editable, and executable PDL programs that use this library. This approach also enables source-to-source optimization, allowing human-in-the-loop refinement and reuse. Evaluations across three tasks and six LLMs (ranging from 3B to 70B parameters) show consistent accuracy gains ($9.06 \pm 15.3$ percentage points), up to 68.9pp, and reveal that selected prompting strategies vary across models and tasks.} }
Endnote
%0 Conference Paper %T AutoPDL: Automatic Prompt Optimization for LLM Agents %A Claudio Spiess %A Mandana Vaziri %A Louis Mandel %A Martin Hirzel %B Proceedings of the Fourth International Conference on Automated Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Leman Akoglu %E Carola Doerr %E Jan N. van Rijn %E Roman Garnett %E Jacob R. Gardner %F pmlr-v293-spiess25a %I PMLR %P 13/1--20 %U https://proceedings.mlr.press/v293/spiess25a.html %V 293 %X The performance of large language models (LLMs) depends on how they are prompted, with choices spanning both the high-level prompting pattern (e.g., Zero-Shot, CoT, ReAct, ReWOO) and the specific prompt content (instructions and few-shot demonstrations). Manually tuning this combination is tedious, error-prone, and non-transferable across LLMs or tasks. Therefore, this paper proposes AutoPDL, an automated approach to discover good LLM agent configurations. Our method frames this as a structured AutoML problem over a combinatorial space of agentic and non-agentic prompting patterns and demonstrations, using successive halving to efficiently navigate this space. We introduce a library implementing common prompting patterns using the PDL prompt programming language. AutoPDL solutions are human-readable, editable, and executable PDL programs that use this library. This approach also enables source-to-source optimization, allowing human-in-the-loop refinement and reuse. Evaluations across three tasks and six LLMs (ranging from 3B to 70B parameters) show consistent accuracy gains ($9.06 \pm 15.3$ percentage points), up to 68.9pp, and reveal that selected prompting strategies vary across models and tasks.
APA
Spiess, C., Vaziri, M., Mandel, L. & Hirzel, M.. (2025). AutoPDL: Automatic Prompt Optimization for LLM Agents. Proceedings of the Fourth International Conference on Automated Machine Learning, in Proceedings of Machine Learning Research 293:13/1-20 Available from https://proceedings.mlr.press/v293/spiess25a.html.

Related Material