Nonlinear low-dimensional regression using auxiliary coordinates

Weiran Wang, Miguel Carreira-Perpinan
; Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, PMLR 22:1295-1304, 2012.

Abstract

When doing regression with inputs and outputs that are high-dimensional, it often makes sense to reduce the dimensionality of the inputs before mapping to the outputs. Much work in statistics and machine learning, such as reduced-rank regression, slice inverse regression and their variants, has focused on linear dimensionality reduction, or on estimating the dimensionality reduction first and then the mapping. We propose a method where both the dimensionality reduction and the mapping can be nonlinear and are estimated jointly. Our key idea is to define an objective function where the low-dimensional coordinates are free parameters, in addition to the dimensionality reduction and the mapping. This has the effect of decoupling many groups of parameters from each other, affording a far more effective optimization than if using a deep network with nested mappings, and to use a good initialization from slice inverse regression or spectral methods. Our experiments with image and robot applications show our approach to improve over direct regression and various existing approaches.

Cite this Paper


BibTeX
@InProceedings{pmlr-v22-wang12, title = {Nonlinear low-dimensional regression using auxiliary coordinates}, author = {Weiran Wang and Miguel Carreira-Perpinan}, pages = {1295--1304}, year = {2012}, editor = {Neil D. Lawrence and Mark Girolami}, volume = {22}, series = {Proceedings of Machine Learning Research}, address = {La Palma, Canary Islands}, month = {21--23 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v22/wang12/wang12.pdf}, url = {http://proceedings.mlr.press/v22/wang12.html}, abstract = {When doing regression with inputs and outputs that are high-dimensional, it often makes sense to reduce the dimensionality of the inputs before mapping to the outputs. Much work in statistics and machine learning, such as reduced-rank regression, slice inverse regression and their variants, has focused on linear dimensionality reduction, or on estimating the dimensionality reduction first and then the mapping. We propose a method where both the dimensionality reduction and the mapping can be nonlinear and are estimated jointly. Our key idea is to define an objective function where the low-dimensional coordinates are free parameters, in addition to the dimensionality reduction and the mapping. This has the effect of decoupling many groups of parameters from each other, affording a far more effective optimization than if using a deep network with nested mappings, and to use a good initialization from slice inverse regression or spectral methods. Our experiments with image and robot applications show our approach to improve over direct regression and various existing approaches.} }
Endnote
%0 Conference Paper %T Nonlinear low-dimensional regression using auxiliary coordinates %A Weiran Wang %A Miguel Carreira-Perpinan %B Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2012 %E Neil D. Lawrence %E Mark Girolami %F pmlr-v22-wang12 %I PMLR %J Proceedings of Machine Learning Research %P 1295--1304 %U http://proceedings.mlr.press %V 22 %W PMLR %X When doing regression with inputs and outputs that are high-dimensional, it often makes sense to reduce the dimensionality of the inputs before mapping to the outputs. Much work in statistics and machine learning, such as reduced-rank regression, slice inverse regression and their variants, has focused on linear dimensionality reduction, or on estimating the dimensionality reduction first and then the mapping. We propose a method where both the dimensionality reduction and the mapping can be nonlinear and are estimated jointly. Our key idea is to define an objective function where the low-dimensional coordinates are free parameters, in addition to the dimensionality reduction and the mapping. This has the effect of decoupling many groups of parameters from each other, affording a far more effective optimization than if using a deep network with nested mappings, and to use a good initialization from slice inverse regression or spectral methods. Our experiments with image and robot applications show our approach to improve over direct regression and various existing approaches.
RIS
TY - CPAPER TI - Nonlinear low-dimensional regression using auxiliary coordinates AU - Weiran Wang AU - Miguel Carreira-Perpinan BT - Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics PY - 2012/03/21 DA - 2012/03/21 ED - Neil D. Lawrence ED - Mark Girolami ID - pmlr-v22-wang12 PB - PMLR SP - 1295 DP - PMLR EP - 1304 L1 - http://proceedings.mlr.press/v22/wang12/wang12.pdf UR - http://proceedings.mlr.press/v22/wang12.html AB - When doing regression with inputs and outputs that are high-dimensional, it often makes sense to reduce the dimensionality of the inputs before mapping to the outputs. Much work in statistics and machine learning, such as reduced-rank regression, slice inverse regression and their variants, has focused on linear dimensionality reduction, or on estimating the dimensionality reduction first and then the mapping. We propose a method where both the dimensionality reduction and the mapping can be nonlinear and are estimated jointly. Our key idea is to define an objective function where the low-dimensional coordinates are free parameters, in addition to the dimensionality reduction and the mapping. This has the effect of decoupling many groups of parameters from each other, affording a far more effective optimization than if using a deep network with nested mappings, and to use a good initialization from slice inverse regression or spectral methods. Our experiments with image and robot applications show our approach to improve over direct regression and various existing approaches. ER -
APA
Wang, W. & Carreira-Perpinan, M.. (2012). Nonlinear low-dimensional regression using auxiliary coordinates. Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, in PMLR 22:1295-1304

Related Material