Nonlinear low-dimensional regression using auxiliary coordinates

Weiran Wang, Miguel Carreira-Perpinan
Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, PMLR 22:1295-1304, 2012.

Abstract

When doing regression with inputs and outputs that are high-dimensional, it often makes sense to reduce the dimensionality of the inputs before mapping to the outputs. Much work in statistics and machine learning, such as reduced-rank regression, slice inverse regression and their variants, has focused on linear dimensionality reduction, or on estimating the dimensionality reduction first and then the mapping. We propose a method where both the dimensionality reduction and the mapping can be nonlinear and are estimated jointly. Our key idea is to define an objective function where the low-dimensional coordinates are free parameters, in addition to the dimensionality reduction and the mapping. This has the effect of decoupling many groups of parameters from each other, affording a far more effective optimization than if using a deep network with nested mappings, and to use a good initialization from slice inverse regression or spectral methods. Our experiments with image and robot applications show our approach to improve over direct regression and various existing approaches.

Cite this Paper

BibTeX
@InProceedings{pmlr-v22-wang12, title = {Nonlinear low-dimensional regression using auxiliary coordinates}, author = {Wang, Weiran and Carreira-Perpinan, Miguel}, booktitle = {Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics}, pages = {1295--1304}, year = {2012}, editor = {Lawrence, Neil D. and Girolami, Mark}, volume = {22}, series = {Proceedings of Machine Learning Research}, address = {La Palma, Canary Islands}, month = {21--23 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v22/wang12/wang12.pdf}, url = {https://proceedings.mlr.press/v22/wang12.html}, abstract = {When doing regression with inputs and outputs that are high-dimensional, it often makes sense to reduce the dimensionality of the inputs before mapping to the outputs. Much work in statistics and machine learning, such as reduced-rank regression, slice inverse regression and their variants, has focused on linear dimensionality reduction, or on estimating the dimensionality reduction first and then the mapping. We propose a method where both the dimensionality reduction and the mapping can be nonlinear and are estimated jointly. Our key idea is to define an objective function where the low-dimensional coordinates are free parameters, in addition to the dimensionality reduction and the mapping. This has the effect of decoupling many groups of parameters from each other, affording a far more effective optimization than if using a deep network with nested mappings, and to use a good initialization from slice inverse regression or spectral methods. Our experiments with image and robot applications show our approach to improve over direct regression and various existing approaches.} }
Endnote
%0 Conference Paper %T Nonlinear low-dimensional regression using auxiliary coordinates %A Weiran Wang %A Miguel Carreira-Perpinan %B Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2012 %E Neil D. Lawrence %E Mark Girolami %F pmlr-v22-wang12 %I PMLR %P 1295--1304 %U https://proceedings.mlr.press/v22/wang12.html %V 22 %X When doing regression with inputs and outputs that are high-dimensional, it often makes sense to reduce the dimensionality of the inputs before mapping to the outputs. Much work in statistics and machine learning, such as reduced-rank regression, slice inverse regression and their variants, has focused on linear dimensionality reduction, or on estimating the dimensionality reduction first and then the mapping. We propose a method where both the dimensionality reduction and the mapping can be nonlinear and are estimated jointly. Our key idea is to define an objective function where the low-dimensional coordinates are free parameters, in addition to the dimensionality reduction and the mapping. This has the effect of decoupling many groups of parameters from each other, affording a far more effective optimization than if using a deep network with nested mappings, and to use a good initialization from slice inverse regression or spectral methods. Our experiments with image and robot applications show our approach to improve over direct regression and various existing approaches.
RIS
TY - CPAPER TI - Nonlinear low-dimensional regression using auxiliary coordinates AU - Weiran Wang AU - Miguel Carreira-Perpinan BT - Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics DA - 2012/03/21 ED - Neil D. Lawrence ED - Mark Girolami ID - pmlr-v22-wang12 PB - PMLR DP - Proceedings of Machine Learning Research VL - 22 SP - 1295 EP - 1304 L1 - http://proceedings.mlr.press/v22/wang12/wang12.pdf UR - https://proceedings.mlr.press/v22/wang12.html AB - When doing regression with inputs and outputs that are high-dimensional, it often makes sense to reduce the dimensionality of the inputs before mapping to the outputs. Much work in statistics and machine learning, such as reduced-rank regression, slice inverse regression and their variants, has focused on linear dimensionality reduction, or on estimating the dimensionality reduction first and then the mapping. We propose a method where both the dimensionality reduction and the mapping can be nonlinear and are estimated jointly. Our key idea is to define an objective function where the low-dimensional coordinates are free parameters, in addition to the dimensionality reduction and the mapping. This has the effect of decoupling many groups of parameters from each other, affording a far more effective optimization than if using a deep network with nested mappings, and to use a good initialization from slice inverse regression or spectral methods. Our experiments with image and robot applications show our approach to improve over direct regression and various existing approaches. ER -
APA
Wang, W. & Carreira-Perpinan, M.. (2012). Nonlinear low-dimensional regression using auxiliary coordinates. Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 22:1295-1304 Available from https://proceedings.mlr.press/v22/wang12.html.

Related Material