AgentHPO: Large Language Model Agent for Hyper-Parameter Optimization

Siyi Liu, Chen Gao, Yong Li
Conference on Parsimony and Learning, PMLR 280:1146-1169, 2025.

Abstract

Hyperparameter optimization is critical in modern machine learning, requiring expert knowledge, numerous trials, and high computational and human resources. Despite the advancements in Automated Machine Learning (AutoML), challenges in terms of trial efficiency, setup complexity, and interoperability still persist. To address these issues, we introduce a novel paradigm leveraging Large Language Models (LLMs) to automate hyperparameter optimization across diverse machine learning tasks, which is named AgentHPO (short for LLM Agent-based Hyper parameter Optimization). Specifically, AgentHPO processes the task information autonomously, conducts experiments with specific hyperparameters (HPs), and iteratively optimizes them based on historical trials. This human-like optimization process largely reduces the number of required trials, simplifies the setup pro cess, and enhances interpretability and user trust, compared to traditional AutoML methods. Extensive empirical experiments conducted on 12 representative machine learning tasks indicate that AgentHPO not only matches but also often surpasses the best human trials in terms of performance while simultaneously providing explainable results. Further analysis sheds light on the strategies employed by the LLMin optimizing these tasks, highlighting its effectiveness and adaptability in various scenarios.

Cite this Paper

BibTeX
@InProceedings{pmlr-v280-liu25c, title = {AgentHPO: Large Language Model Agent for Hyper-Parameter Optimization}, author = {Liu, Siyi and Gao, Chen and Li, Yong}, booktitle = {Conference on Parsimony and Learning}, pages = {1146--1169}, year = {2025}, editor = {Chen, Beidi and Liu, Shijia and Pilanci, Mert and Su, Weijie and Sulam, Jeremias and Wang, Yuxiang and Zhu, Zhihui}, volume = {280}, series = {Proceedings of Machine Learning Research}, month = {24--27 Mar}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v280/main/assets/liu25c/liu25c.pdf}, url = {https://proceedings.mlr.press/v280/liu25c.html}, abstract = {Hyperparameter optimization is critical in modern machine learning, requiring expert knowledge, numerous trials, and high computational and human resources. Despite the advancements in Automated Machine Learning (AutoML), challenges in terms of trial efficiency, setup complexity, and interoperability still persist. To address these issues, we introduce a novel paradigm leveraging Large Language Models (LLMs) to automate hyperparameter optimization across diverse machine learning tasks, which is named AgentHPO (short for LLM Agent-based Hyper parameter Optimization). Specifically, AgentHPO processes the task information autonomously, conducts experiments with specific hyperparameters (HPs), and iteratively optimizes them based on historical trials. This human-like optimization process largely reduces the number of required trials, simplifies the setup pro cess, and enhances interpretability and user trust, compared to traditional AutoML methods. Extensive empirical experiments conducted on 12 representative machine learning tasks indicate that AgentHPO not only matches but also often surpasses the best human trials in terms of performance while simultaneously providing explainable results. Further analysis sheds light on the strategies employed by the LLMin optimizing these tasks, highlighting its effectiveness and adaptability in various scenarios.} }
Endnote
%0 Conference Paper %T AgentHPO: Large Language Model Agent for Hyper-Parameter Optimization %A Siyi Liu %A Chen Gao %A Yong Li %B Conference on Parsimony and Learning %C Proceedings of Machine Learning Research %D 2025 %E Beidi Chen %E Shijia Liu %E Mert Pilanci %E Weijie Su %E Jeremias Sulam %E Yuxiang Wang %E Zhihui Zhu %F pmlr-v280-liu25c %I PMLR %P 1146--1169 %U https://proceedings.mlr.press/v280/liu25c.html %V 280 %X Hyperparameter optimization is critical in modern machine learning, requiring expert knowledge, numerous trials, and high computational and human resources. Despite the advancements in Automated Machine Learning (AutoML), challenges in terms of trial efficiency, setup complexity, and interoperability still persist. To address these issues, we introduce a novel paradigm leveraging Large Language Models (LLMs) to automate hyperparameter optimization across diverse machine learning tasks, which is named AgentHPO (short for LLM Agent-based Hyper parameter Optimization). Specifically, AgentHPO processes the task information autonomously, conducts experiments with specific hyperparameters (HPs), and iteratively optimizes them based on historical trials. This human-like optimization process largely reduces the number of required trials, simplifies the setup pro cess, and enhances interpretability and user trust, compared to traditional AutoML methods. Extensive empirical experiments conducted on 12 representative machine learning tasks indicate that AgentHPO not only matches but also often surpasses the best human trials in terms of performance while simultaneously providing explainable results. Further analysis sheds light on the strategies employed by the LLMin optimizing these tasks, highlighting its effectiveness and adaptability in various scenarios.
APA
Liu, S., Gao, C. & Li, Y.. (2025). AgentHPO: Large Language Model Agent for Hyper-Parameter Optimization. Conference on Parsimony and Learning, in Proceedings of Machine Learning Research 280:1146-1169 Available from https://proceedings.mlr.press/v280/liu25c.html.

Related Material