High-dimensional Joint Sparsity Random Effects Model for Multi-task Learning

Krishnakumar Balasubramanian, Kai Yu, Tong Zhang
Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence, PMLR R11:282-291, 2013.

Abstract

Joint sparsity regularization in multi-task learning has attracted much attention in recent years. The traditional convex formulation employs the group Lasso relaxation to achieve joint sparsity across tasks. Although this approach leads to a simple convex formulation, it suffers from several issues due to the looseness of the relaxation. To remedy this problem, we view jointly sparse multi-task learning as a specialized random effects model, and derive a convex relaxation approach that involves two steps. The first step learns the covariance matrix of the coefficients using a convex formulation which we refer to as sparse covariance coding; the second step solves a ridge regression problem with a sparse quadratic regularizer based on the covariance matrix obtained in the first step. It is shown that this approach produces an asymptotically optimal quadratic regularizer in the multitask learning setting when the number of tasks approaches infinity. Experimental results demonstrate that the convex formulation obtained via the proposed model significantly outperforms group Lasso (and related multi-stage formulations).

Cite this Paper

BibTeX
@InProceedings{pmlr-vR11-balasubramanian13a, title = {High-dimensional Joint Sparsity Random Effects Model for Multi-task Learning}, author = {Balasubramanian, Krishnakumar and Yu, Kai and Zhang, Tong}, booktitle = {Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence}, pages = {282--291}, year = {2013}, editor = {Nicholson, Ann and Smyth, Padhraic}, volume = {R11}, series = {Proceedings of Machine Learning Research}, month = {12--14 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/r11/main/assets/balasubramanian13a/balasubramanian13a.pdf}, url = {https://proceedings.mlr.press/r11/balasubramanian13a.html}, abstract = {Joint sparsity regularization in multi-task learning has attracted much attention in recent years. The traditional convex formulation employs the group Lasso relaxation to achieve joint sparsity across tasks. Although this approach leads to a simple convex formulation, it suffers from several issues due to the looseness of the relaxation. To remedy this problem, we view jointly sparse multi-task learning as a specialized random effects model, and derive a convex relaxation approach that involves two steps. The first step learns the covariance matrix of the coefficients using a convex formulation which we refer to as sparse covariance coding; the second step solves a ridge regression problem with a sparse quadratic regularizer based on the covariance matrix obtained in the first step. It is shown that this approach produces an asymptotically optimal quadratic regularizer in the multitask learning setting when the number of tasks approaches infinity. Experimental results demonstrate that the convex formulation obtained via the proposed model significantly outperforms group Lasso (and related multi-stage formulations).}, note = {Reissued by PMLR on 04 October 2026.} }
Endnote
%0 Conference Paper %T High-dimensional Joint Sparsity Random Effects Model for Multi-task Learning %A Krishnakumar Balasubramanian %A Kai Yu %A Tong Zhang %B Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2013 %E Ann Nicholson %E Padhraic Smyth %F pmlr-vR11-balasubramanian13a %I PMLR %P 282--291 %U https://proceedings.mlr.press/r11/balasubramanian13a.html %V R11 %X Joint sparsity regularization in multi-task learning has attracted much attention in recent years. The traditional convex formulation employs the group Lasso relaxation to achieve joint sparsity across tasks. Although this approach leads to a simple convex formulation, it suffers from several issues due to the looseness of the relaxation. To remedy this problem, we view jointly sparse multi-task learning as a specialized random effects model, and derive a convex relaxation approach that involves two steps. The first step learns the covariance matrix of the coefficients using a convex formulation which we refer to as sparse covariance coding; the second step solves a ridge regression problem with a sparse quadratic regularizer based on the covariance matrix obtained in the first step. It is shown that this approach produces an asymptotically optimal quadratic regularizer in the multitask learning setting when the number of tasks approaches infinity. Experimental results demonstrate that the convex formulation obtained via the proposed model significantly outperforms group Lasso (and related multi-stage formulations). %Z Reissued by PMLR on 04 October 2026.
APA
Balasubramanian, K., Yu, K. & Zhang, T.. (2013). High-dimensional Joint Sparsity Random Effects Model for Multi-task Learning. Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research R11:282-291 Available from https://proceedings.mlr.press/r11/balasubramanian13a.html. Reissued by PMLR on 04 October 2026.

Related Material