A Unified Energy-Based Framework for Unsupervised Learning

Marc’Aurelio Ranzato, Y-Lan Boureau, Sumit Chopra, Yann LeCun
Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, PMLR 2:371-379, 2007.

Abstract

We introduce a view of unsupervised learning that integrates probabilistic and nonprobabilistic methods for clustering, dimensionality reduction, and feature extraction in a unified framework. In this framework, an energy function associates low energies to input points that are similar to training samples, and high energies to unobserved points. Learning consists in minimizing the energies of training samples while ensuring that the energies of unobserved ones are higher. Some traditional methods construct the architecture so that only a small number of points can have low energy, while other methods explicitly “pull up” on the energies of unobserved points. In probabilistic methods the energy of unobserved points is pulled by minimizing the log partition function, an expensive, and sometimes intractable process. We explore different and more efficient methods using an energy-based approach. In particular, we show that a simple solution is to restrict the amount of information contained in codes that represent the data. We demonstrate such a method by training it on natural image patches and by applying to image denoising.

Cite this Paper

BibTeX
@InProceedings{pmlr-v2-ranzato07a, title = {A Unified Energy-Based Framework for Unsupervised Learning}, author = {Ranzato, Marc’Aurelio and Boureau, Y-Lan and Chopra, Sumit and LeCun, Yann}, booktitle = {Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics}, pages = {371--379}, year = {2007}, editor = {Meila, Marina and Shen, Xiaotong}, volume = {2}, series = {Proceedings of Machine Learning Research}, address = {San Juan, Puerto Rico}, month = {21--24 Mar}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v2/ranzato07a/ranzato07a.pdf}, url = {https://proceedings.mlr.press/v2/ranzato07a.html}, abstract = {We introduce a view of unsupervised learning that integrates probabilistic and nonprobabilistic methods for clustering, dimensionality reduction, and feature extraction in a unified framework. In this framework, an energy function associates low energies to input points that are similar to training samples, and high energies to unobserved points. Learning consists in minimizing the energies of training samples while ensuring that the energies of unobserved ones are higher. Some traditional methods construct the architecture so that only a small number of points can have low energy, while other methods explicitly “pull up” on the energies of unobserved points. In probabilistic methods the energy of unobserved points is pulled by minimizing the log partition function, an expensive, and sometimes intractable process. We explore different and more efficient methods using an energy-based approach. In particular, we show that a simple solution is to restrict the amount of information contained in codes that represent the data. We demonstrate such a method by training it on natural image patches and by applying to image denoising.} }
Endnote
%0 Conference Paper %T A Unified Energy-Based Framework for Unsupervised Learning %A Marc’Aurelio Ranzato %A Y-Lan Boureau %A Sumit Chopra %A Yann LeCun %B Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2007 %E Marina Meila %E Xiaotong Shen %F pmlr-v2-ranzato07a %I PMLR %P 371--379 %U https://proceedings.mlr.press/v2/ranzato07a.html %V 2 %X We introduce a view of unsupervised learning that integrates probabilistic and nonprobabilistic methods for clustering, dimensionality reduction, and feature extraction in a unified framework. In this framework, an energy function associates low energies to input points that are similar to training samples, and high energies to unobserved points. Learning consists in minimizing the energies of training samples while ensuring that the energies of unobserved ones are higher. Some traditional methods construct the architecture so that only a small number of points can have low energy, while other methods explicitly “pull up” on the energies of unobserved points. In probabilistic methods the energy of unobserved points is pulled by minimizing the log partition function, an expensive, and sometimes intractable process. We explore different and more efficient methods using an energy-based approach. In particular, we show that a simple solution is to restrict the amount of information contained in codes that represent the data. We demonstrate such a method by training it on natural image patches and by applying to image denoising.
RIS
TY - CPAPER TI - A Unified Energy-Based Framework for Unsupervised Learning AU - Marc’Aurelio Ranzato AU - Y-Lan Boureau AU - Sumit Chopra AU - Yann LeCun BT - Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics DA - 2007/03/11 ED - Marina Meila ED - Xiaotong Shen ID - pmlr-v2-ranzato07a PB - PMLR DP - Proceedings of Machine Learning Research VL - 2 SP - 371 EP - 379 L1 - http://proceedings.mlr.press/v2/ranzato07a/ranzato07a.pdf UR - https://proceedings.mlr.press/v2/ranzato07a.html AB - We introduce a view of unsupervised learning that integrates probabilistic and nonprobabilistic methods for clustering, dimensionality reduction, and feature extraction in a unified framework. In this framework, an energy function associates low energies to input points that are similar to training samples, and high energies to unobserved points. Learning consists in minimizing the energies of training samples while ensuring that the energies of unobserved ones are higher. Some traditional methods construct the architecture so that only a small number of points can have low energy, while other methods explicitly “pull up” on the energies of unobserved points. In probabilistic methods the energy of unobserved points is pulled by minimizing the log partition function, an expensive, and sometimes intractable process. We explore different and more efficient methods using an energy-based approach. In particular, we show that a simple solution is to restrict the amount of information contained in codes that represent the data. We demonstrate such a method by training it on natural image patches and by applying to image denoising. ER -
APA
Ranzato, M., Boureau, Y., Chopra, S. & LeCun, Y.. (2007). A Unified Energy-Based Framework for Unsupervised Learning. Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 2:371-379 Available from https://proceedings.mlr.press/v2/ranzato07a.html.

Related Material