When, where, and how to add new neurons to ANNs

Kaitlin Maile, Emmanuel Rachelson, Hervé Luga, Dennis George Wilson
Proceedings of the First International Conference on Automated Machine Learning, PMLR 188:18/1-12, 2022.

Abstract

Neurogenesis in ANNs is an understudied and difficult problem, even compared to other forms of structural learning like pruning. By decomposing it into triggers and initializations, we introduce a framework for studying the various facets of neurogenesis: when, where, and how to add neurons during the learning process. We present the Neural Orthogonality (NORTH*) suite of neurogenesis strategies, combining layer-wise triggers and initializations based on the orthogonality of activations or weights to dynamically grow performant networks that converge to an efficient size. We evaluate our contributions against other recent neurogenesis works across a variety of supervised learning tasks.

Cite this Paper

BibTeX
@InProceedings{pmlr-v188-maile22a, title = {When, where, and how to add new neurons to ANNs}, author = {Maile, Kaitlin and Rachelson, Emmanuel and Luga, Herv\'e and Wilson, Dennis George}, booktitle = {Proceedings of the First International Conference on Automated Machine Learning}, pages = {18/1--12}, year = {2022}, editor = {Guyon, Isabelle and Lindauer, Marius and van der Schaar, Mihaela and Hutter, Frank and Garnett, Roman}, volume = {188}, series = {Proceedings of Machine Learning Research}, month = {25--27 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v188/maile22a/maile22a.pdf}, url = {https://proceedings.mlr.press/v188/maile22a.html}, abstract = {Neurogenesis in ANNs is an understudied and difficult problem, even compared to other forms of structural learning like pruning. By decomposing it into triggers and initializations, we introduce a framework for studying the various facets of neurogenesis: when, where, and how to add neurons during the learning process. We present the Neural Orthogonality (NORTH*) suite of neurogenesis strategies, combining layer-wise triggers and initializations based on the orthogonality of activations or weights to dynamically grow performant networks that converge to an efficient size. We evaluate our contributions against other recent neurogenesis works across a variety of supervised learning tasks.} }
Endnote
%0 Conference Paper %T When, where, and how to add new neurons to ANNs %A Kaitlin Maile %A Emmanuel Rachelson %A Hervé Luga %A Dennis George Wilson %B Proceedings of the First International Conference on Automated Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Isabelle Guyon %E Marius Lindauer %E Mihaela van der Schaar %E Frank Hutter %E Roman Garnett %F pmlr-v188-maile22a %I PMLR %P 18/1--12 %U https://proceedings.mlr.press/v188/maile22a.html %V 188 %X Neurogenesis in ANNs is an understudied and difficult problem, even compared to other forms of structural learning like pruning. By decomposing it into triggers and initializations, we introduce a framework for studying the various facets of neurogenesis: when, where, and how to add neurons during the learning process. We present the Neural Orthogonality (NORTH*) suite of neurogenesis strategies, combining layer-wise triggers and initializations based on the orthogonality of activations or weights to dynamically grow performant networks that converge to an efficient size. We evaluate our contributions against other recent neurogenesis works across a variety of supervised learning tasks.
APA
Maile, K., Rachelson, E., Luga, H. & Wilson, D.G.. (2022). When, where, and how to add new neurons to ANNs. Proceedings of the First International Conference on Automated Machine Learning, in Proceedings of Machine Learning Research 188:18/1-12 Available from https://proceedings.mlr.press/v188/maile22a.html.

Related Material