Towards Coherent and Consistent Use of Entities in Narrative Generation

Pinelopi Papalampidi, Kris Cao, Tomas Kocisky
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:17278-17294, 2022.

Abstract

Large pre-trained language models (LMs) have demonstrated impressive capabilities in generating long, fluent text; however, there is little to no analysis on their ability to maintain entity coherence and consistency. In this work, we focus on the end task of narrative generation and systematically analyse the long-range entity coherence and consistency in generated stories. First, we propose a set of automatic metrics for measuring model performance in terms of entity usage. Given these metrics, we quantify the limitations of current LMs. Next, we propose augmenting a pre-trained LM with a dynamic entity memory in an end-to-end manner by using an auxiliary entity-related loss for guiding the reads and writes to the memory. We demonstrate that the dynamic entity memory increases entity coherence according to both automatic and human judgment and helps preserving entity-related information especially in settings with a limited context window. Finally, we also validate that our automatic metrics are correlated with human ratings and serve as a good indicator of the quality of generated stories.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-papalampidi22a, title = {Towards Coherent and Consistent Use of Entities in Narrative Generation}, author = {Papalampidi, Pinelopi and Cao, Kris and Kocisky, Tomas}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {17278--17294}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/papalampidi22a/papalampidi22a.pdf}, url = {https://proceedings.mlr.press/v162/papalampidi22a.html}, abstract = {Large pre-trained language models (LMs) have demonstrated impressive capabilities in generating long, fluent text; however, there is little to no analysis on their ability to maintain entity coherence and consistency. In this work, we focus on the end task of narrative generation and systematically analyse the long-range entity coherence and consistency in generated stories. First, we propose a set of automatic metrics for measuring model performance in terms of entity usage. Given these metrics, we quantify the limitations of current LMs. Next, we propose augmenting a pre-trained LM with a dynamic entity memory in an end-to-end manner by using an auxiliary entity-related loss for guiding the reads and writes to the memory. We demonstrate that the dynamic entity memory increases entity coherence according to both automatic and human judgment and helps preserving entity-related information especially in settings with a limited context window. Finally, we also validate that our automatic metrics are correlated with human ratings and serve as a good indicator of the quality of generated stories.} }
Endnote
%0 Conference Paper %T Towards Coherent and Consistent Use of Entities in Narrative Generation %A Pinelopi Papalampidi %A Kris Cao %A Tomas Kocisky %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-papalampidi22a %I PMLR %P 17278--17294 %U https://proceedings.mlr.press/v162/papalampidi22a.html %V 162 %X Large pre-trained language models (LMs) have demonstrated impressive capabilities in generating long, fluent text; however, there is little to no analysis on their ability to maintain entity coherence and consistency. In this work, we focus on the end task of narrative generation and systematically analyse the long-range entity coherence and consistency in generated stories. First, we propose a set of automatic metrics for measuring model performance in terms of entity usage. Given these metrics, we quantify the limitations of current LMs. Next, we propose augmenting a pre-trained LM with a dynamic entity memory in an end-to-end manner by using an auxiliary entity-related loss for guiding the reads and writes to the memory. We demonstrate that the dynamic entity memory increases entity coherence according to both automatic and human judgment and helps preserving entity-related information especially in settings with a limited context window. Finally, we also validate that our automatic metrics are correlated with human ratings and serve as a good indicator of the quality of generated stories.
APA
Papalampidi, P., Cao, K. & Kocisky, T.. (2022). Towards Coherent and Consistent Use of Entities in Narrative Generation. Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:17278-17294 Available from https://proceedings.mlr.press/v162/papalampidi22a.html.

Related Material