Supervised and Semi-Supervised Text Categorization using LSTM for Region Embeddings

Rie Johnson, Tong Zhang
Proceedings of The 33rd International Conference on Machine Learning, PMLR 48:526-534, 2016.

Abstract

One-hot CNN (convolutional neural network) has been shown to be effective for text categorization (Johnson & Zhang, 2015). We view it as a special case of a general framework which jointly trains a linear model with a non-linear feature generator consisting of ‘text region embedding + pooling’. Under this framework, we explore a more sophisticated region embedding method using Long Short-Term Memory (LSTM). LSTM can embed text regions of variable (and possibly large) sizes, whereas the region size needs to be fixed in a CNN. We seek effective and efficient use of LSTM for this purpose in the supervised and semi-supervised settings. The best results were obtained by combining region embeddings in the form of LSTM and convolution layers trained on unlabeled data. The results indicate that on this task, embeddings of text regions, which can convey complex concepts, are more useful than embeddings of single words in isolation. We report performances exceeding the previous best results on four benchmark datasets.

Cite this Paper

BibTeX
@InProceedings{pmlr-v48-johnson16, title = {Supervised and Semi-Supervised Text Categorization using LSTM for Region Embeddings}, author = {Johnson, Rie and Zhang, Tong}, booktitle = {Proceedings of The 33rd International Conference on Machine Learning}, pages = {526--534}, year = {2016}, editor = {Balcan, Maria Florina and Weinberger, Kilian Q.}, volume = {48}, series = {Proceedings of Machine Learning Research}, address = {New York, New York, USA}, month = {20--22 Jun}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v48/johnson16.pdf}, url = {https://proceedings.mlr.press/v48/johnson16.html}, abstract = {One-hot CNN (convolutional neural network) has been shown to be effective for text categorization (Johnson & Zhang, 2015). We view it as a special case of a general framework which jointly trains a linear model with a non-linear feature generator consisting of ‘text region embedding + pooling’. Under this framework, we explore a more sophisticated region embedding method using Long Short-Term Memory (LSTM). LSTM can embed text regions of variable (and possibly large) sizes, whereas the region size needs to be fixed in a CNN. We seek effective and efficient use of LSTM for this purpose in the supervised and semi-supervised settings. The best results were obtained by combining region embeddings in the form of LSTM and convolution layers trained on unlabeled data. The results indicate that on this task, embeddings of text regions, which can convey complex concepts, are more useful than embeddings of single words in isolation. We report performances exceeding the previous best results on four benchmark datasets.} }
Endnote
%0 Conference Paper %T Supervised and Semi-Supervised Text Categorization using LSTM for Region Embeddings %A Rie Johnson %A Tong Zhang %B Proceedings of The 33rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2016 %E Maria Florina Balcan %E Kilian Q. Weinberger %F pmlr-v48-johnson16 %I PMLR %P 526--534 %U https://proceedings.mlr.press/v48/johnson16.html %V 48 %X One-hot CNN (convolutional neural network) has been shown to be effective for text categorization (Johnson & Zhang, 2015). We view it as a special case of a general framework which jointly trains a linear model with a non-linear feature generator consisting of ‘text region embedding + pooling’. Under this framework, we explore a more sophisticated region embedding method using Long Short-Term Memory (LSTM). LSTM can embed text regions of variable (and possibly large) sizes, whereas the region size needs to be fixed in a CNN. We seek effective and efficient use of LSTM for this purpose in the supervised and semi-supervised settings. The best results were obtained by combining region embeddings in the form of LSTM and convolution layers trained on unlabeled data. The results indicate that on this task, embeddings of text regions, which can convey complex concepts, are more useful than embeddings of single words in isolation. We report performances exceeding the previous best results on four benchmark datasets.
RIS
TY - CPAPER TI - Supervised and Semi-Supervised Text Categorization using LSTM for Region Embeddings AU - Rie Johnson AU - Tong Zhang BT - Proceedings of The 33rd International Conference on Machine Learning DA - 2016/06/11 ED - Maria Florina Balcan ED - Kilian Q. Weinberger ID - pmlr-v48-johnson16 PB - PMLR DP - Proceedings of Machine Learning Research VL - 48 SP - 526 EP - 534 L1 - http://proceedings.mlr.press/v48/johnson16.pdf UR - https://proceedings.mlr.press/v48/johnson16.html AB - One-hot CNN (convolutional neural network) has been shown to be effective for text categorization (Johnson & Zhang, 2015). We view it as a special case of a general framework which jointly trains a linear model with a non-linear feature generator consisting of ‘text region embedding + pooling’. Under this framework, we explore a more sophisticated region embedding method using Long Short-Term Memory (LSTM). LSTM can embed text regions of variable (and possibly large) sizes, whereas the region size needs to be fixed in a CNN. We seek effective and efficient use of LSTM for this purpose in the supervised and semi-supervised settings. The best results were obtained by combining region embeddings in the form of LSTM and convolution layers trained on unlabeled data. The results indicate that on this task, embeddings of text regions, which can convey complex concepts, are more useful than embeddings of single words in isolation. We report performances exceeding the previous best results on four benchmark datasets. ER -
APA
Johnson, R. & Zhang, T.. (2016). Supervised and Semi-Supervised Text Categorization using LSTM for Region Embeddings. Proceedings of The 33rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 48:526-534 Available from https://proceedings.mlr.press/v48/johnson16.html.

Related Material