A new evaluation framework for topic modeling algorithms based on synthetic corpora

Hanyu Shi, Martin Gerlach, Isabel Diersen, Doug Downey, Luis Amaral
Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, PMLR 89:816-826, 2019.

Abstract

Topic models are in widespread use in natural language processing and beyond. Here, we propose a new framework for the evaluation of topic modeling algorithms based on synthetic corpora containing an unambiguously defined ground truth topic structure. The major innovation of our approach is the ability to quantify the agreement between the planted and inferred topic structures by comparing the assigned topic labels at the level of the tokens. In experiments, our approach yields novel insights about the relative strengths of topic models as corpus characteristics vary, and the first evidence of an “undetectable phase” for topic models when the planted structure is weak. We also establish the practical relevance of the insights gained for synthetic corpora by predicting the performance of topic modeling algorithms in classification tasks in real-world corpora.

Cite this Paper

BibTeX
@InProceedings{pmlr-v89-shi19a, title = {A new evaluation framework for topic modeling algorithms based on synthetic corpora}, author = {Shi, Hanyu and Gerlach, Martin and Diersen, Isabel and Downey, Doug and Amaral, Luis}, booktitle = {Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics}, pages = {816--826}, year = {2019}, editor = {Chaudhuri, Kamalika and Sugiyama, Masashi}, volume = {89}, series = {Proceedings of Machine Learning Research}, month = {16--18 Apr}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v89/shi19a/shi19a.pdf}, url = {https://proceedings.mlr.press/v89/shi19a.html}, abstract = {Topic models are in widespread use in natural language processing and beyond. Here, we propose a new framework for the evaluation of topic modeling algorithms based on synthetic corpora containing an unambiguously defined ground truth topic structure. The major innovation of our approach is the ability to quantify the agreement between the planted and inferred topic structures by comparing the assigned topic labels at the level of the tokens. In experiments, our approach yields novel insights about the relative strengths of topic models as corpus characteristics vary, and the first evidence of an “undetectable phase” for topic models when the planted structure is weak. We also establish the practical relevance of the insights gained for synthetic corpora by predicting the performance of topic modeling algorithms in classification tasks in real-world corpora.} }
Endnote
%0 Conference Paper %T A new evaluation framework for topic modeling algorithms based on synthetic corpora %A Hanyu Shi %A Martin Gerlach %A Isabel Diersen %A Doug Downey %A Luis Amaral %B Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2019 %E Kamalika Chaudhuri %E Masashi Sugiyama %F pmlr-v89-shi19a %I PMLR %P 816--826 %U https://proceedings.mlr.press/v89/shi19a.html %V 89 %X Topic models are in widespread use in natural language processing and beyond. Here, we propose a new framework for the evaluation of topic modeling algorithms based on synthetic corpora containing an unambiguously defined ground truth topic structure. The major innovation of our approach is the ability to quantify the agreement between the planted and inferred topic structures by comparing the assigned topic labels at the level of the tokens. In experiments, our approach yields novel insights about the relative strengths of topic models as corpus characteristics vary, and the first evidence of an “undetectable phase” for topic models when the planted structure is weak. We also establish the practical relevance of the insights gained for synthetic corpora by predicting the performance of topic modeling algorithms in classification tasks in real-world corpora.
APA
Shi, H., Gerlach, M., Diersen, I., Downey, D. & Amaral, L.. (2019). A new evaluation framework for topic modeling algorithms based on synthetic corpora. Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 89:816-826 Available from https://proceedings.mlr.press/v89/shi19a.html.

Related Material