Learning Dynamics of Linear Denoising Autoencoders

Arnu Pretorius, Steve Kroon, Herman Kamper
Proceedings of the 35th International Conference on Machine Learning, PMLR 80:4141-4150, 2018.

Abstract

Denoising autoencoders (DAEs) have proven useful for unsupervised representation learning, but a thorough theoretical understanding is still lacking of how the input noise influences learning. Here we develop theory for how noise influences learning in DAEs. By focusing on linear DAEs, we are able to derive analytic expressions that exactly describe their learning dynamics. We verify our theoretical predictions with simulations as well as experiments on MNIST and CIFAR-10. The theory illustrates how, when tuned correctly, noise allows DAEs to ignore low variance directions in the inputs while learning to reconstruct them. Furthermore, in a comparison of the learning dynamics of DAEs to standard regularised autoencoders, we show that noise has a similar regularisation effect to weight decay, but with faster training dynamics. We also show that our theoretical predictions approximate learning dynamics on real-world data and qualitatively match observed dynamics in nonlinear DAEs.

Cite this Paper

BibTeX
@InProceedings{pmlr-v80-pretorius18a, title = {Learning Dynamics of Linear Denoising Autoencoders}, author = {Pretorius, Arnu and Kroon, Steve and Kamper, Herman}, booktitle = {Proceedings of the 35th International Conference on Machine Learning}, pages = {4141--4150}, year = {2018}, editor = {Dy, Jennifer and Krause, Andreas}, volume = {80}, series = {Proceedings of Machine Learning Research}, month = {10--15 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v80/pretorius18a/pretorius18a.pdf}, url = {https://proceedings.mlr.press/v80/pretorius18a.html}, abstract = {Denoising autoencoders (DAEs) have proven useful for unsupervised representation learning, but a thorough theoretical understanding is still lacking of how the input noise influences learning. Here we develop theory for how noise influences learning in DAEs. By focusing on linear DAEs, we are able to derive analytic expressions that exactly describe their learning dynamics. We verify our theoretical predictions with simulations as well as experiments on MNIST and CIFAR-10. The theory illustrates how, when tuned correctly, noise allows DAEs to ignore low variance directions in the inputs while learning to reconstruct them. Furthermore, in a comparison of the learning dynamics of DAEs to standard regularised autoencoders, we show that noise has a similar regularisation effect to weight decay, but with faster training dynamics. We also show that our theoretical predictions approximate learning dynamics on real-world data and qualitatively match observed dynamics in nonlinear DAEs.} }
Endnote
%0 Conference Paper %T Learning Dynamics of Linear Denoising Autoencoders %A Arnu Pretorius %A Steve Kroon %A Herman Kamper %B Proceedings of the 35th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2018 %E Jennifer Dy %E Andreas Krause %F pmlr-v80-pretorius18a %I PMLR %P 4141--4150 %U https://proceedings.mlr.press/v80/pretorius18a.html %V 80 %X Denoising autoencoders (DAEs) have proven useful for unsupervised representation learning, but a thorough theoretical understanding is still lacking of how the input noise influences learning. Here we develop theory for how noise influences learning in DAEs. By focusing on linear DAEs, we are able to derive analytic expressions that exactly describe their learning dynamics. We verify our theoretical predictions with simulations as well as experiments on MNIST and CIFAR-10. The theory illustrates how, when tuned correctly, noise allows DAEs to ignore low variance directions in the inputs while learning to reconstruct them. Furthermore, in a comparison of the learning dynamics of DAEs to standard regularised autoencoders, we show that noise has a similar regularisation effect to weight decay, but with faster training dynamics. We also show that our theoretical predictions approximate learning dynamics on real-world data and qualitatively match observed dynamics in nonlinear DAEs.
APA
Pretorius, A., Kroon, S. & Kamper, H.. (2018). Learning Dynamics of Linear Denoising Autoencoders. Proceedings of the 35th International Conference on Machine Learning, in Proceedings of Machine Learning Research 80:4141-4150 Available from https://proceedings.mlr.press/v80/pretorius18a.html.

Related Material