Improving Transfer Learning by means of Ensemble Learning and Swarm Intelligence-based Neuroevolution

Adri Gómez, Monica Abella, Manuel Desco
Proceedings of the Third International Conference on Automated Machine Learning, PMLR 256:7/1-25, 2024.

Abstract

Neural Architecture Search (NAS) methods, when applied to very small but complex datasets, tend to overfit on the validation partitions and underperform compared to Transfer Learning models. In order to reduce the bias and variance of their predictions, Deep Ensemble Learning (DEL) can be used. The combination of NAS and DEL has only been employed on large datasets in the literature, but these scenarios do not present the overfitting in validation we typically experience, for instance, on medical imaging applications. In this work, we empirically assess the feasibility of NAS, DEL and the combination of the two on both small and large dataset scenarios. We find that the performance of the ensembles highly depend on the degree of overfitting of the standalone models, but always will outperform the worst generalizing models in the population.

Cite this Paper

BibTeX
@InProceedings{pmlr-v256-gomez24a, title = {Improving Transfer Learning by means of Ensemble Learning and Swarm Intelligence-based Neuroevolution}, author = {G\'omez, Adri and Abella, Monica and Desco, Manuel}, booktitle = {Proceedings of the Third International Conference on Automated Machine Learning}, pages = {7/1--25}, year = {2024}, editor = {Eggensperger, Katharina and Garnett, Roman and Vanschoren, Joaquin and Lindauer, Marius and Gardner, Jacob R.}, volume = {256}, series = {Proceedings of Machine Learning Research}, month = {09--12 Sep}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v256/main/assets/gomez24a/gomez24a.pdf}, url = {https://proceedings.mlr.press/v256/gomez24a.html}, abstract = {Neural Architecture Search (NAS) methods, when applied to very small but complex datasets, tend to overfit on the validation partitions and underperform compared to Transfer Learning models. In order to reduce the bias and variance of their predictions, Deep Ensemble Learning (DEL) can be used. The combination of NAS and DEL has only been employed on large datasets in the literature, but these scenarios do not present the overfitting in validation we typically experience, for instance, on medical imaging applications. In this work, we empirically assess the feasibility of NAS, DEL and the combination of the two on both small and large dataset scenarios. We find that the performance of the ensembles highly depend on the degree of overfitting of the standalone models, but always will outperform the worst generalizing models in the population.} }
Endnote
%0 Conference Paper %T Improving Transfer Learning by means of Ensemble Learning and Swarm Intelligence-based Neuroevolution %A Adri Gómez %A Monica Abella %A Manuel Desco %B Proceedings of the Third International Conference on Automated Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Katharina Eggensperger %E Roman Garnett %E Joaquin Vanschoren %E Marius Lindauer %E Jacob R. Gardner %F pmlr-v256-gomez24a %I PMLR %P 7/1--25 %U https://proceedings.mlr.press/v256/gomez24a.html %V 256 %X Neural Architecture Search (NAS) methods, when applied to very small but complex datasets, tend to overfit on the validation partitions and underperform compared to Transfer Learning models. In order to reduce the bias and variance of their predictions, Deep Ensemble Learning (DEL) can be used. The combination of NAS and DEL has only been employed on large datasets in the literature, but these scenarios do not present the overfitting in validation we typically experience, for instance, on medical imaging applications. In this work, we empirically assess the feasibility of NAS, DEL and the combination of the two on both small and large dataset scenarios. We find that the performance of the ensembles highly depend on the degree of overfitting of the standalone models, but always will outperform the worst generalizing models in the population.
APA
Gómez, A., Abella, M. & Desco, M.. (2024). Improving Transfer Learning by means of Ensemble Learning and Swarm Intelligence-based Neuroevolution. Proceedings of the Third International Conference on Automated Machine Learning, in Proceedings of Machine Learning Research 256:7/1-25 Available from https://proceedings.mlr.press/v256/gomez24a.html.

Related Material