On the effectiveness of the skew divergence for statistical language analysis

Lillian Lee
Proceedings of the Eighth International Workshop on Artificial Intelligence and Statistics, PMLR R3:176-183, 2001.

Abstract

Estimating word co-occurrence probabilities is a problem underlying many applications in statistical natural language processing. Distance-weighted (or similarityweighted) averaging has been shown to be a promising approach to the analysis of novel co-occurrences. Many measures of distributional similarity have been proposed for use in the distance-weighted averaging framework; here, we empirically study their stability properties, finding that similarity-based estimation appears to make more efficient use of more reliable portions of the training data. We also investigate properties of the skew divergence, a weighted version of the KullbackLeibler (KL) divergence; our results indicate that the skew divergence yields better results than the KL divergence even when the KL divergence is applied to more sophisticated probability estimates.

Cite this Paper

BibTeX
@InProceedings{pmlr-vR3-lee01a, title = {On the effectiveness of the skew divergence for statistical language analysis}, author = {Lee, Lillian}, booktitle = {Proceedings of the Eighth International Workshop on Artificial Intelligence and Statistics}, pages = {176--183}, year = {2001}, editor = {Richardson, Thomas S. and Jaakkola, Tommi S.}, volume = {R3}, series = {Proceedings of Machine Learning Research}, month = {04--07 Jan}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/r3/lee01a/lee01a.pdf}, url = {https://proceedings.mlr.press/r3/lee01a.html}, abstract = {Estimating word co-occurrence probabilities is a problem underlying many applications in statistical natural language processing. Distance-weighted (or similarityweighted) averaging has been shown to be a promising approach to the analysis of novel co-occurrences. Many measures of distributional similarity have been proposed for use in the distance-weighted averaging framework; here, we empirically study their stability properties, finding that similarity-based estimation appears to make more efficient use of more reliable portions of the training data. We also investigate properties of the skew divergence, a weighted version of the KullbackLeibler (KL) divergence; our results indicate that the skew divergence yields better results than the KL divergence even when the KL divergence is applied to more sophisticated probability estimates.}, note = {Reissued by PMLR on 31 March 2021.} }
Endnote
%0 Conference Paper %T On the effectiveness of the skew divergence for statistical language analysis %A Lillian Lee %B Proceedings of the Eighth International Workshop on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2001 %E Thomas S. Richardson %E Tommi S. Jaakkola %F pmlr-vR3-lee01a %I PMLR %P 176--183 %U https://proceedings.mlr.press/r3/lee01a.html %V R3 %X Estimating word co-occurrence probabilities is a problem underlying many applications in statistical natural language processing. Distance-weighted (or similarityweighted) averaging has been shown to be a promising approach to the analysis of novel co-occurrences. Many measures of distributional similarity have been proposed for use in the distance-weighted averaging framework; here, we empirically study their stability properties, finding that similarity-based estimation appears to make more efficient use of more reliable portions of the training data. We also investigate properties of the skew divergence, a weighted version of the KullbackLeibler (KL) divergence; our results indicate that the skew divergence yields better results than the KL divergence even when the KL divergence is applied to more sophisticated probability estimates. %Z Reissued by PMLR on 31 March 2021.
APA
Lee, L.. (2001). On the effectiveness of the skew divergence for statistical language analysis. Proceedings of the Eighth International Workshop on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research R3:176-183 Available from https://proceedings.mlr.press/r3/lee01a.html. Reissued by PMLR on 31 March 2021.

Related Material