Fully Parallel Hyperparameter Search: Reshaped Space-Filling

Marie-Liesse Cauwet, Camille Couprie, Julien Dehos, Pauline Luc, Jeremy Rapin, Morgane Riviere, Fabien Teytaud, Olivier Teytaud, Nicolas Usunier
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:1338-1348, 2020.

Abstract

Space-filling designs such as Low Discrepancy Sequence (LDS), Latin Hypercube Sampling (LHS) and Jittered Sampling (JS) were proposed for fully parallel hyperparameter search, and were shown to be more effective than random and grid search. We prove that LHS and JS outperform random search only by a constant factor. Consequently, we introduce a new sampling approach based on the reshaping of the search distribution, and we show both theoretically and numerically that it leads to significant gains over random search. Two methods are proposed for the reshaping: Recentering (when the distribution of the optimum is known), and Cauchy transformation (when the distribution of the optimum is unknown). The proposed methods are first validated on artificial experiments and simple real-world tests on clustering and Salmon mappings. Then we demonstrate that they drive performance improvement in a wide range of expensive artificial intelligence tasks, namely attend/infer/repeat, video next frame segmentation forecasting and progressive generative adversarial networks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-cauwet20a, title = {Fully Parallel Hyperparameter Search: Reshaped Space-Filling}, author = {Cauwet, Marie-Liesse and Couprie, Camille and Dehos, Julien and Luc, Pauline and Rapin, Jeremy and Riviere, Morgane and Teytaud, Fabien and Teytaud, Olivier and Usunier, Nicolas}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {1338--1348}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/cauwet20a/cauwet20a.pdf}, url = {https://proceedings.mlr.press/v119/cauwet20a.html}, abstract = {Space-filling designs such as Low Discrepancy Sequence (LDS), Latin Hypercube Sampling (LHS) and Jittered Sampling (JS) were proposed for fully parallel hyperparameter search, and were shown to be more effective than random and grid search. We prove that LHS and JS outperform random search only by a constant factor. Consequently, we introduce a new sampling approach based on the reshaping of the search distribution, and we show both theoretically and numerically that it leads to significant gains over random search. Two methods are proposed for the reshaping: Recentering (when the distribution of the optimum is known), and Cauchy transformation (when the distribution of the optimum is unknown). The proposed methods are first validated on artificial experiments and simple real-world tests on clustering and Salmon mappings. Then we demonstrate that they drive performance improvement in a wide range of expensive artificial intelligence tasks, namely attend/infer/repeat, video next frame segmentation forecasting and progressive generative adversarial networks.} }
Endnote
%0 Conference Paper %T Fully Parallel Hyperparameter Search: Reshaped Space-Filling %A Marie-Liesse Cauwet %A Camille Couprie %A Julien Dehos %A Pauline Luc %A Jeremy Rapin %A Morgane Riviere %A Fabien Teytaud %A Olivier Teytaud %A Nicolas Usunier %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-cauwet20a %I PMLR %P 1338--1348 %U https://proceedings.mlr.press/v119/cauwet20a.html %V 119 %X Space-filling designs such as Low Discrepancy Sequence (LDS), Latin Hypercube Sampling (LHS) and Jittered Sampling (JS) were proposed for fully parallel hyperparameter search, and were shown to be more effective than random and grid search. We prove that LHS and JS outperform random search only by a constant factor. Consequently, we introduce a new sampling approach based on the reshaping of the search distribution, and we show both theoretically and numerically that it leads to significant gains over random search. Two methods are proposed for the reshaping: Recentering (when the distribution of the optimum is known), and Cauchy transformation (when the distribution of the optimum is unknown). The proposed methods are first validated on artificial experiments and simple real-world tests on clustering and Salmon mappings. Then we demonstrate that they drive performance improvement in a wide range of expensive artificial intelligence tasks, namely attend/infer/repeat, video next frame segmentation forecasting and progressive generative adversarial networks.
APA
Cauwet, M., Couprie, C., Dehos, J., Luc, P., Rapin, J., Riviere, M., Teytaud, F., Teytaud, O. & Usunier, N.. (2020). Fully Parallel Hyperparameter Search: Reshaped Space-Filling. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:1338-1348 Available from https://proceedings.mlr.press/v119/cauwet20a.html.

Related Material