Annealing Paths for the Evaluation of Topic Models

Statistical topic models such as latent Dirich- let allocation have become enormously popu- lar in the past decade, with dozens of learning algorithms and extensions being proposed each year. As these models and algorithms continue to be developed, it becomes increasingly impor- tant to evaluate them relative to previous tech- niques. However, evaluating the predictive per- formance of a topic model is a computationally difficult task. Annealed importance sampling (AIS), a Monte Carlo technique which operates by annealing between two distributions, has pre- viously been successfully used for topic model evaluation (Wallach et al., 2009b). This tech- nique estimates the likelihood of a held-out doc- ument by simulating an annealing process from the prior to the posterior for the latent topic as- signments, and using this simulation as an im- portance sampling proposal distribution. In this paper we introduce new AIS annealing paths which instead anneal from one topic model to another, thereby estimating the relative perfor- mance of the models. This strategy can exhibit much lower empirical variance than previous ap- proaches, facilitating reliable per-documentcom- parisons of topic models. We then show how to use these paths to evaluate the predictive perfor- mance of topic model learning algorithms by effi- ciently estimating the likelihood at each iteration of the training procedure. The proposed method achieves better held-out likelihood estimates for this task than previous algorithms with, in some cases, an order of magnitude less computation.