Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree

Chen-Yu Lee, Patrick W. Gallagher, Zhuowen Tu
Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, PMLR 51:464-472, 2016.

Abstract

We seek to improve deep neural networks by generalizing the pooling operations that play a central role in current architectures. We pursue a careful exploration of approaches to allow pooling to learn and to adapt to complex and variable patterns. The two primary directions lie in (1) learning a pooling function via (two strategies of) combining of max and average pooling, and (2) learning a pooling function in the form of a tree-structured fusion of pooling filters that are themselves learned. In our experiments every generalized pooling operation we explore improves performance when used in place of average or max pooling. We experimentally demonstrate that the proposed pooling operations provide a boost in invariance properties relative to conventional pooling and set the state of the art on several widely adopted benchmark datasets; they are also easy to implement, and can be applied within various deep neural network architectures. These benefits come with only a light increase in computational overhead during training and a very modest increase in the number of model parameters.

Cite this Paper

BibTeX
@InProceedings{pmlr-v51-lee16a, title = {Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree}, author = {Lee, Chen-Yu and Gallagher, Patrick W. and Tu, Zhuowen}, booktitle = {Proceedings of the 19th International Conference on Artificial Intelligence and Statistics}, pages = {464--472}, year = {2016}, editor = {Gretton, Arthur and Robert, Christian C.}, volume = {51}, series = {Proceedings of Machine Learning Research}, address = {Cadiz, Spain}, month = {09--11 May}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v51/lee16a.pdf}, url = {https://proceedings.mlr.press/v51/lee16a.html}, abstract = {We seek to improve deep neural networks by generalizing the pooling operations that play a central role in current architectures. We pursue a careful exploration of approaches to allow pooling to learn and to adapt to complex and variable patterns. The two primary directions lie in (1) learning a pooling function via (two strategies of) combining of max and average pooling, and (2) learning a pooling function in the form of a tree-structured fusion of pooling filters that are themselves learned. In our experiments every generalized pooling operation we explore improves performance when used in place of average or max pooling. We experimentally demonstrate that the proposed pooling operations provide a boost in invariance properties relative to conventional pooling and set the state of the art on several widely adopted benchmark datasets; they are also easy to implement, and can be applied within various deep neural network architectures. These benefits come with only a light increase in computational overhead during training and a very modest increase in the number of model parameters.} }
Endnote
%0 Conference Paper %T Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree %A Chen-Yu Lee %A Patrick W. Gallagher %A Zhuowen Tu %B Proceedings of the 19th International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2016 %E Arthur Gretton %E Christian C. Robert %F pmlr-v51-lee16a %I PMLR %P 464--472 %U https://proceedings.mlr.press/v51/lee16a.html %V 51 %X We seek to improve deep neural networks by generalizing the pooling operations that play a central role in current architectures. We pursue a careful exploration of approaches to allow pooling to learn and to adapt to complex and variable patterns. The two primary directions lie in (1) learning a pooling function via (two strategies of) combining of max and average pooling, and (2) learning a pooling function in the form of a tree-structured fusion of pooling filters that are themselves learned. In our experiments every generalized pooling operation we explore improves performance when used in place of average or max pooling. We experimentally demonstrate that the proposed pooling operations provide a boost in invariance properties relative to conventional pooling and set the state of the art on several widely adopted benchmark datasets; they are also easy to implement, and can be applied within various deep neural network architectures. These benefits come with only a light increase in computational overhead during training and a very modest increase in the number of model parameters.
RIS
TY - CPAPER TI - Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree AU - Chen-Yu Lee AU - Patrick W. Gallagher AU - Zhuowen Tu BT - Proceedings of the 19th International Conference on Artificial Intelligence and Statistics DA - 2016/05/02 ED - Arthur Gretton ED - Christian C. Robert ID - pmlr-v51-lee16a PB - PMLR DP - Proceedings of Machine Learning Research VL - 51 SP - 464 EP - 472 L1 - http://proceedings.mlr.press/v51/lee16a.pdf UR - https://proceedings.mlr.press/v51/lee16a.html AB - We seek to improve deep neural networks by generalizing the pooling operations that play a central role in current architectures. We pursue a careful exploration of approaches to allow pooling to learn and to adapt to complex and variable patterns. The two primary directions lie in (1) learning a pooling function via (two strategies of) combining of max and average pooling, and (2) learning a pooling function in the form of a tree-structured fusion of pooling filters that are themselves learned. In our experiments every generalized pooling operation we explore improves performance when used in place of average or max pooling. We experimentally demonstrate that the proposed pooling operations provide a boost in invariance properties relative to conventional pooling and set the state of the art on several widely adopted benchmark datasets; they are also easy to implement, and can be applied within various deep neural network architectures. These benefits come with only a light increase in computational overhead during training and a very modest increase in the number of model parameters. ER -
APA
Lee, C., Gallagher, P.W. & Tu, Z.. (2016). Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree. Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 51:464-472 Available from https://proceedings.mlr.press/v51/lee16a.html.

Related Material