Duality in RKHSs with Infinite Dimensional Outputs: Application to Robust Losses

Pierre Laforgue, Alex Lambert, Luc Brogat-Motte, Florence D’Alché-Buc
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:5598-5607, 2020.

Abstract

Operator-Valued Kernels (OVKs) and associated vector-valued Reproducing Kernel Hilbert Spaces provide an elegant way to extend scalar kernel methods when the output space is a Hilbert space. Although primarily used in finite dimension for problems like multi-task regression, the ability of this framework to deal with infinite dimensional output spaces unlocks many more applications, such as functional regression, structured output prediction, and structured data representation. However, these sophisticated schemes crucially rely on the kernel trick in the output space, so that most of previous works have focused on the square norm loss function, completely neglecting robustness issues that may arise in such surrogate problems. To overcome this limitation, this paper develops a duality approach that allows to solve OVK machines for a wide range of loss functions. The infinite dimensional Lagrange multipliers are handled through a Double Representer Theorem, and algorithms for \epsilon-insensitive losses and the Huber loss are thoroughly detailed. Robustness benefits are emphasized by a theoretical stability analysis, as well as empirical improvements on structured data applications.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-laforgue20a, title = {Duality in {RKHS}s with Infinite Dimensional Outputs: Application to Robust Losses}, author = {Laforgue, Pierre and Lambert, Alex and Brogat-Motte, Luc and D'Alch{\'e}-Buc, Florence}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {5598--5607}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/laforgue20a/laforgue20a.pdf}, url = {https://proceedings.mlr.press/v119/laforgue20a.html}, abstract = {Operator-Valued Kernels (OVKs) and associated vector-valued Reproducing Kernel Hilbert Spaces provide an elegant way to extend scalar kernel methods when the output space is a Hilbert space. Although primarily used in finite dimension for problems like multi-task regression, the ability of this framework to deal with infinite dimensional output spaces unlocks many more applications, such as functional regression, structured output prediction, and structured data representation. However, these sophisticated schemes crucially rely on the kernel trick in the output space, so that most of previous works have focused on the square norm loss function, completely neglecting robustness issues that may arise in such surrogate problems. To overcome this limitation, this paper develops a duality approach that allows to solve OVK machines for a wide range of loss functions. The infinite dimensional Lagrange multipliers are handled through a Double Representer Theorem, and algorithms for \epsilon-insensitive losses and the Huber loss are thoroughly detailed. Robustness benefits are emphasized by a theoretical stability analysis, as well as empirical improvements on structured data applications.} }
Endnote
%0 Conference Paper %T Duality in RKHSs with Infinite Dimensional Outputs: Application to Robust Losses %A Pierre Laforgue %A Alex Lambert %A Luc Brogat-Motte %A Florence D’Alché-Buc %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-laforgue20a %I PMLR %P 5598--5607 %U https://proceedings.mlr.press/v119/laforgue20a.html %V 119 %X Operator-Valued Kernels (OVKs) and associated vector-valued Reproducing Kernel Hilbert Spaces provide an elegant way to extend scalar kernel methods when the output space is a Hilbert space. Although primarily used in finite dimension for problems like multi-task regression, the ability of this framework to deal with infinite dimensional output spaces unlocks many more applications, such as functional regression, structured output prediction, and structured data representation. However, these sophisticated schemes crucially rely on the kernel trick in the output space, so that most of previous works have focused on the square norm loss function, completely neglecting robustness issues that may arise in such surrogate problems. To overcome this limitation, this paper develops a duality approach that allows to solve OVK machines for a wide range of loss functions. The infinite dimensional Lagrange multipliers are handled through a Double Representer Theorem, and algorithms for \epsilon-insensitive losses and the Huber loss are thoroughly detailed. Robustness benefits are emphasized by a theoretical stability analysis, as well as empirical improvements on structured data applications.
APA
Laforgue, P., Lambert, A., Brogat-Motte, L. & D’Alché-Buc, F.. (2020). Duality in RKHSs with Infinite Dimensional Outputs: Application to Robust Losses. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:5598-5607 Available from https://proceedings.mlr.press/v119/laforgue20a.html.

Related Material