Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selection

Matteo Zecchin, Sangwoo Park, Osvaldo Simeone
Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:74018-74036, 2025.

Abstract

We introduce adaptive learn-then-test (aLTT), an efficient hyperparameter selection procedure that provides finite-sample statistical guarantees on the population risk of AI models. Unlike the existing learn-then-test (LTT) technique, which relies on conventional p-value-based multiple hypothesis testing (MHT), aLTT implements sequential data-dependent MHT with early termination by leveraging e-processes. As a result, aLTT can reduce the number of testing rounds, making it particularly well-suited for scenarios in which testing is costly or presents safety risks. Apart from maintaining statistical validity, in applications such as online policy selection for offline reinforcement learning and prompt engineering, aLTT is shown to achieve the same performance as LTT while requiring only a fraction of the testing rounds.

Cite this Paper

BibTeX
@InProceedings{pmlr-v267-zecchin25a, title = {Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selection}, author = {Zecchin, Matteo and Park, Sangwoo and Simeone, Osvaldo}, booktitle = {Proceedings of the 42nd International Conference on Machine Learning}, pages = {74018--74036}, year = {2025}, editor = {Singh, Aarti and Fazel, Maryam and Hsu, Daniel and Lacoste-Julien, Simon and Berkenkamp, Felix and Maharaj, Tegan and Wagstaff, Kiri and Zhu, Jerry}, volume = {267}, series = {Proceedings of Machine Learning Research}, month = {13--19 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v267/main/assets/zecchin25a/zecchin25a.pdf}, url = {https://proceedings.mlr.press/v267/zecchin25a.html}, abstract = {We introduce adaptive learn-then-test (aLTT), an efficient hyperparameter selection procedure that provides finite-sample statistical guarantees on the population risk of AI models. Unlike the existing learn-then-test (LTT) technique, which relies on conventional p-value-based multiple hypothesis testing (MHT), aLTT implements sequential data-dependent MHT with early termination by leveraging e-processes. As a result, aLTT can reduce the number of testing rounds, making it particularly well-suited for scenarios in which testing is costly or presents safety risks. Apart from maintaining statistical validity, in applications such as online policy selection for offline reinforcement learning and prompt engineering, aLTT is shown to achieve the same performance as LTT while requiring only a fraction of the testing rounds.} }
Endnote
%0 Conference Paper %T Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selection %A Matteo Zecchin %A Sangwoo Park %A Osvaldo Simeone %B Proceedings of the 42nd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2025 %E Aarti Singh %E Maryam Fazel %E Daniel Hsu %E Simon Lacoste-Julien %E Felix Berkenkamp %E Tegan Maharaj %E Kiri Wagstaff %E Jerry Zhu %F pmlr-v267-zecchin25a %I PMLR %P 74018--74036 %U https://proceedings.mlr.press/v267/zecchin25a.html %V 267 %X We introduce adaptive learn-then-test (aLTT), an efficient hyperparameter selection procedure that provides finite-sample statistical guarantees on the population risk of AI models. Unlike the existing learn-then-test (LTT) technique, which relies on conventional p-value-based multiple hypothesis testing (MHT), aLTT implements sequential data-dependent MHT with early termination by leveraging e-processes. As a result, aLTT can reduce the number of testing rounds, making it particularly well-suited for scenarios in which testing is costly or presents safety risks. Apart from maintaining statistical validity, in applications such as online policy selection for offline reinforcement learning and prompt engineering, aLTT is shown to achieve the same performance as LTT while requiring only a fraction of the testing rounds.
APA
Zecchin, M., Park, S. & Simeone, O.. (2025). Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selection. Proceedings of the 42nd International Conference on Machine Learning, in Proceedings of Machine Learning Research 267:74018-74036 Available from https://proceedings.mlr.press/v267/zecchin25a.html.

Related Material