Latent Maximum Entropy Approach for Semantic $N$-gram Language Modeling

Shaojun Wang, Dale Schuurmans, Fuchun Peng
Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics, PMLR R4:316-322, 2003.

Abstract

In this paper, we describe a unified probabilistic framework for statistical language modeling-the latent maximum entropy principle-which can effectively incorporate various aspects of natural language, such as local word interaction, syntactic structure and semantic document information. Unlike previous work on maximum entropy methods for language modeling, which only allow explicit features to be modeled, our framework also allows relationships over hidden features to be captured, resulting in a more expressive language model. We describe efficient algorithms for marginalization, inference and normalization in our extended models. We then present promising experimental results for our approach on the Wall Street Journal corpus.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR4-wang03a, title = {Latent Maximum Entropy Approach for Semantic $N$-gram Language Modeling}, author = {Wang, Shaojun and Schuurmans, Dale and Peng, Fuchun}, booktitle = {Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics}, pages = {316--322}, year = {2003}, editor = {Bishop, Christopher M. and Frey, Brendan J.}, volume = {R4}, series = {Proceedings of Machine Learning Research}, month = {03--06 Jan}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/r4/wang03a/wang03a.pdf}, url = {https://proceedings.mlr.press/r4/wang03a.html}, abstract = {In this paper, we describe a unified probabilistic framework for statistical language modeling-the latent maximum entropy principle-which can effectively incorporate various aspects of natural language, such as local word interaction, syntactic structure and semantic document information. Unlike previous work on maximum entropy methods for language modeling, which only allow explicit features to be modeled, our framework also allows relationships over hidden features to be captured, resulting in a more expressive language model. We describe efficient algorithms for marginalization, inference and normalization in our extended models. We then present promising experimental results for our approach on the Wall Street Journal corpus.}, note = {Reissued by PMLR on 01 April 2021.} }
Endnote
%0 Conference Paper %T Latent Maximum Entropy Approach for Semantic $N$-gram Language Modeling %A Shaojun Wang %A Dale Schuurmans %A Fuchun Peng %B Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2003 %E Christopher M. Bishop %E Brendan J. Frey %F pmlr-vR4-wang03a %I PMLR %P 316--322 %U https://proceedings.mlr.press/r4/wang03a.html %V R4 %X In this paper, we describe a unified probabilistic framework for statistical language modeling-the latent maximum entropy principle-which can effectively incorporate various aspects of natural language, such as local word interaction, syntactic structure and semantic document information. Unlike previous work on maximum entropy methods for language modeling, which only allow explicit features to be modeled, our framework also allows relationships over hidden features to be captured, resulting in a more expressive language model. We describe efficient algorithms for marginalization, inference and normalization in our extended models. We then present promising experimental results for our approach on the Wall Street Journal corpus. %Z Reissued by PMLR on 01 April 2021.
APA
Wang, S., Schuurmans, D. & Peng, F.. (2003). Latent Maximum Entropy Approach for Semantic $N$-gram Language Modeling. Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research R4:316-322 Available from https://proceedings.mlr.press/r4/wang03a.html. Reissued by PMLR on 01 April 2021.

Related Material