Speeding up NAS with Adaptive Subset Selection

Vishak Prasad C, Colin White, Sibasis Nayak, Paarth Jain, Aziz Shameem, Prateek Garg, Ganesh Ramakrishnan
Proceedings of the Third International Conference on Automated Machine Learning, PMLR 256:3/1-23, 2024.

Abstract

A majority of recent developments in neural architecture search (NAS) have been aimed at decreasing the computational cost of various techniques without affecting their final performance. Towards this goal, several low-fidelity and performance prediction methods have been considered, including those that train only on subsets of the training data. In this work, we present an adaptive subset selection approach to NAS and present it as complementary to state-of-the-art NAS approaches. We uncover a natural connection between one-shot NAS algorithms and adaptive subset selection and devise an algorithm that makes use of state-of-the-art techniques from both areas. We use these techniques to substantially reduce the runtime of DARTS-PT (a leading one-shot NAS algorithm), as well as BOHB and DEHB (leading multi-fidelity optimization algorithms), with minimal sacrifice to accuracy. In experiments, we find architectures on CIFAR-10 that give 5% increase in performance over DARTS-PT while reducing the time required by more than 8 times. Our results are consistent across multiple datasets, and towards full reproducibility, we release all our code at \url{https://anonymous.4open.science/r/SubsetSelection_NAS-87B3}.

Cite this Paper

BibTeX
@InProceedings{pmlr-v256-c24a, title = {Speeding up NAS with Adaptive Subset Selection}, author = {C, Vishak Prasad and White, Colin and Nayak, Sibasis and Jain, Paarth and Shameem, Aziz and Garg, Prateek and Ramakrishnan, Ganesh}, booktitle = {Proceedings of the Third International Conference on Automated Machine Learning}, pages = {3/1--23}, year = {2024}, editor = {Eggensperger, Katharina and Garnett, Roman and Vanschoren, Joaquin and Lindauer, Marius and Gardner, Jacob R.}, volume = {256}, series = {Proceedings of Machine Learning Research}, month = {09--12 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v256/main/assets/c24a/c24a.pdf}, url = {https://proceedings.mlr.press/v256/c24a.html}, abstract = {A majority of recent developments in neural architecture search (NAS) have been aimed at decreasing the computational cost of various techniques without affecting their final performance. Towards this goal, several low-fidelity and performance prediction methods have been considered, including those that train only on subsets of the training data. In this work, we present an adaptive subset selection approach to NAS and present it as complementary to state-of-the-art NAS approaches. We uncover a natural connection between one-shot NAS algorithms and adaptive subset selection and devise an algorithm that makes use of state-of-the-art techniques from both areas. We use these techniques to substantially reduce the runtime of DARTS-PT (a leading one-shot NAS algorithm), as well as BOHB and DEHB (leading multi-fidelity optimization algorithms), with minimal sacrifice to accuracy. In experiments, we find architectures on CIFAR-10 that give 5% increase in performance over DARTS-PT while reducing the time required by more than 8 times. Our results are consistent across multiple datasets, and towards full reproducibility, we release all our code at \url{https://anonymous.4open.science/r/SubsetSelection_NAS-87B3}.} }
Endnote
%0 Conference Paper %T Speeding up NAS with Adaptive Subset Selection %A Vishak Prasad C %A Colin White %A Sibasis Nayak %A Paarth Jain %A Aziz Shameem %A Prateek Garg %A Ganesh Ramakrishnan %B Proceedings of the Third International Conference on Automated Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Katharina Eggensperger %E Roman Garnett %E Joaquin Vanschoren %E Marius Lindauer %E Jacob R. Gardner %F pmlr-v256-c24a %I PMLR %P 3/1--23 %U https://proceedings.mlr.press/v256/c24a.html %V 256 %X A majority of recent developments in neural architecture search (NAS) have been aimed at decreasing the computational cost of various techniques without affecting their final performance. Towards this goal, several low-fidelity and performance prediction methods have been considered, including those that train only on subsets of the training data. In this work, we present an adaptive subset selection approach to NAS and present it as complementary to state-of-the-art NAS approaches. We uncover a natural connection between one-shot NAS algorithms and adaptive subset selection and devise an algorithm that makes use of state-of-the-art techniques from both areas. We use these techniques to substantially reduce the runtime of DARTS-PT (a leading one-shot NAS algorithm), as well as BOHB and DEHB (leading multi-fidelity optimization algorithms), with minimal sacrifice to accuracy. In experiments, we find architectures on CIFAR-10 that give 5% increase in performance over DARTS-PT while reducing the time required by more than 8 times. Our results are consistent across multiple datasets, and towards full reproducibility, we release all our code at \url{https://anonymous.4open.science/r/SubsetSelection_NAS-87B3}.
APA
C, V.P., White, C., Nayak, S., Jain, P., Shameem, A., Garg, P. & Ramakrishnan, G.. (2024). Speeding up NAS with Adaptive Subset Selection. Proceedings of the Third International Conference on Automated Machine Learning, in Proceedings of Machine Learning Research 256:3/1-23 Available from https://proceedings.mlr.press/v256/c24a.html.

Related Material