Deep Learners Benefit More from Out-of-Distribution Examples

Yoshua Bengio, Frédéric Bastien, Arnaud Bergeron, Nicolas Boulanger–Lewandowski, Thomas Breuel, Youssouf Chherawala, Moustapha Cisse, Myriam Côté, Dumitru Erhan, Jeremy Eustache, Xavier Glorot, Xavier Muller, Sylvain Pannetier Lebeuf, Razvan Pascanu, Salah Rifai, François Savard, Guillaume Sicard
Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, PMLR 15:164-172, 2011.

Abstract

Recent theoretical and empirical work in statistical machine learning has demonstrated the potential of learning algorithms for deep architectures, i.e., function classes obtained by composing multiple levels of representation. The hypothesis evaluated here is that intermediate levels of representation, because they can be shared across tasks and examples from different but related distributions, can yield even more benefits. Comparative experiments were performed on a large-scale handwritten character recognition setting with 62 classes (upper case, lower case, digits), using both a multi-task setting and perturbed examples in order to obtain out-of-distribution examples. The results agree with the hypothesis, and show that a deep learner did beat previously published results and reached human-level performance.

Cite this Paper

BibTeX
@InProceedings{pmlr-v15-bengio11b, title = {Deep Learners Benefit More from Out-of-Distribution Examples}, author = {Bengio, Yoshua and Bastien, Frédéric and Bergeron, Arnaud and Boulanger–Lewandowski, Nicolas and Breuel, Thomas and Chherawala, Youssouf and Cisse, Moustapha and Côté, Myriam and Erhan, Dumitru and Eustache, Jeremy and Glorot, Xavier and Muller, Xavier and Pannetier Lebeuf, Sylvain and Pascanu, Razvan and Rifai, Salah and Savard, François and Sicard, Guillaume}, booktitle = {Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics}, pages = {164--172}, year = {2011}, editor = {Gordon, Geoffrey and Dunson, David and Dudík, Miroslav}, volume = {15}, series = {Proceedings of Machine Learning Research}, address = {Fort Lauderdale, FL, USA}, month = {11--13 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v15/bengio11b/bengio11b.pdf}, url = {https://proceedings.mlr.press/v15/bengio11b.html}, abstract = {Recent theoretical and empirical work in statistical machine learning has demonstrated the potential of learning algorithms for deep architectures, i.e., function classes obtained by composing multiple levels of representation. The hypothesis evaluated here is that intermediate levels of representation, because they can be shared across tasks and examples from different but related distributions, can yield even more benefits. Comparative experiments were performed on a large-scale handwritten character recognition setting with 62 classes (upper case, lower case, digits), using both a multi-task setting and perturbed examples in order to obtain out-of-distribution examples. The results agree with the hypothesis, and show that a deep learner did beat previously published results and reached human-level performance.} }
Endnote
%0 Conference Paper %T Deep Learners Benefit More from Out-of-Distribution Examples %A Yoshua Bengio %A Frédéric Bastien %A Arnaud Bergeron %A Nicolas Boulanger–Lewandowski %A Thomas Breuel %A Youssouf Chherawala %A Moustapha Cisse %A Myriam Côté %A Dumitru Erhan %A Jeremy Eustache %A Xavier Glorot %A Xavier Muller %A Sylvain Pannetier Lebeuf %A Razvan Pascanu %A Salah Rifai %A François Savard %A Guillaume Sicard %B Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2011 %E Geoffrey Gordon %E David Dunson %E Miroslav Dudík %F pmlr-v15-bengio11b %I PMLR %P 164--172 %U https://proceedings.mlr.press/v15/bengio11b.html %V 15 %X Recent theoretical and empirical work in statistical machine learning has demonstrated the potential of learning algorithms for deep architectures, i.e., function classes obtained by composing multiple levels of representation. The hypothesis evaluated here is that intermediate levels of representation, because they can be shared across tasks and examples from different but related distributions, can yield even more benefits. Comparative experiments were performed on a large-scale handwritten character recognition setting with 62 classes (upper case, lower case, digits), using both a multi-task setting and perturbed examples in order to obtain out-of-distribution examples. The results agree with the hypothesis, and show that a deep learner did beat previously published results and reached human-level performance.
RIS
TY - CPAPER TI - Deep Learners Benefit More from Out-of-Distribution Examples AU - Yoshua Bengio AU - Frédéric Bastien AU - Arnaud Bergeron AU - Nicolas Boulanger–Lewandowski AU - Thomas Breuel AU - Youssouf Chherawala AU - Moustapha Cisse AU - Myriam Côté AU - Dumitru Erhan AU - Jeremy Eustache AU - Xavier Glorot AU - Xavier Muller AU - Sylvain Pannetier Lebeuf AU - Razvan Pascanu AU - Salah Rifai AU - François Savard AU - Guillaume Sicard BT - Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics DA - 2011/06/14 ED - Geoffrey Gordon ED - David Dunson ED - Miroslav Dudík ID - pmlr-v15-bengio11b PB - PMLR DP - Proceedings of Machine Learning Research VL - 15 SP - 164 EP - 172 L1 - http://proceedings.mlr.press/v15/bengio11b/bengio11b.pdf UR - https://proceedings.mlr.press/v15/bengio11b.html AB - Recent theoretical and empirical work in statistical machine learning has demonstrated the potential of learning algorithms for deep architectures, i.e., function classes obtained by composing multiple levels of representation. The hypothesis evaluated here is that intermediate levels of representation, because they can be shared across tasks and examples from different but related distributions, can yield even more benefits. Comparative experiments were performed on a large-scale handwritten character recognition setting with 62 classes (upper case, lower case, digits), using both a multi-task setting and perturbed examples in order to obtain out-of-distribution examples. The results agree with the hypothesis, and show that a deep learner did beat previously published results and reached human-level performance. ER -
APA
Bengio, Y., Bastien, F., Bergeron, A., Boulanger–Lewandowski, N., Breuel, T., Chherawala, Y., Cisse, M., Côté, M., Erhan, D., Eustache, J., Glorot, X., Muller, X., Pannetier Lebeuf, S., Pascanu, R., Rifai, S., Savard, F. & Sicard, G.. (2011). Deep Learners Benefit More from Out-of-Distribution Examples. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 15:164-172 Available from https://proceedings.mlr.press/v15/bengio11b.html.

Related Material