Group-Sparse Matrix Factorization for Transfer Learning of Word Embeddings

Kan Xu, Xuanyi Zhao, Hamsa Bastani, Osbert Bastani
Proceedings of the 38th International Conference on Machine Learning, PMLR 139:11603-11612, 2021.

Abstract

Sparse regression has recently been applied to enable transfer learning from very limited data. We study an extension of this approach to unsupervised learning—in particular, learning word embeddings from unstructured text corpora using low-rank matrix factorization. Intuitively, when transferring word embeddings to a new domain, we expect that the embeddings change for only a small number of words—e.g., the ones with novel meanings in that domain. We propose a novel group-sparse penalty that exploits this sparsity to perform transfer learning when there is very little text data available in the target domain—e.g., a single article of text. We prove generalization bounds for our algorithm. Furthermore, we empirically evaluate its effectiveness, both in terms of prediction accuracy in downstream tasks as well as in terms of interpretability of the results.

Cite this Paper

BibTeX
@InProceedings{pmlr-v139-xu21l, title = {Group-Sparse Matrix Factorization for Transfer Learning of Word Embeddings}, author = {Xu, Kan and Zhao, Xuanyi and Bastani, Hamsa and Bastani, Osbert}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {11603--11612}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/xu21l/xu21l.pdf}, url = {https://proceedings.mlr.press/v139/xu21l.html}, abstract = {Sparse regression has recently been applied to enable transfer learning from very limited data. We study an extension of this approach to unsupervised learning—in particular, learning word embeddings from unstructured text corpora using low-rank matrix factorization. Intuitively, when transferring word embeddings to a new domain, we expect that the embeddings change for only a small number of words—e.g., the ones with novel meanings in that domain. We propose a novel group-sparse penalty that exploits this sparsity to perform transfer learning when there is very little text data available in the target domain—e.g., a single article of text. We prove generalization bounds for our algorithm. Furthermore, we empirically evaluate its effectiveness, both in terms of prediction accuracy in downstream tasks as well as in terms of interpretability of the results.} }
Endnote
%0 Conference Paper %T Group-Sparse Matrix Factorization for Transfer Learning of Word Embeddings %A Kan Xu %A Xuanyi Zhao %A Hamsa Bastani %A Osbert Bastani %B Proceedings of the 38th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2021 %E Marina Meila %E Tong Zhang %F pmlr-v139-xu21l %I PMLR %P 11603--11612 %U https://proceedings.mlr.press/v139/xu21l.html %V 139 %X Sparse regression has recently been applied to enable transfer learning from very limited data. We study an extension of this approach to unsupervised learning—in particular, learning word embeddings from unstructured text corpora using low-rank matrix factorization. Intuitively, when transferring word embeddings to a new domain, we expect that the embeddings change for only a small number of words—e.g., the ones with novel meanings in that domain. We propose a novel group-sparse penalty that exploits this sparsity to perform transfer learning when there is very little text data available in the target domain—e.g., a single article of text. We prove generalization bounds for our algorithm. Furthermore, we empirically evaluate its effectiveness, both in terms of prediction accuracy in downstream tasks as well as in terms of interpretability of the results.
APA
Xu, K., Zhao, X., Bastani, H. & Bastani, O.. (2021). Group-Sparse Matrix Factorization for Transfer Learning of Word Embeddings. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:11603-11612 Available from https://proceedings.mlr.press/v139/xu21l.html.

Related Material