Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer

Yanshuai Cao, Peng Xu
Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, PMLR 108:3991-4001, 2020.

Abstract

In this work, we develop a novel regularizer to improve the learning of long-range dependency of sequence data. Applied on language modelling, our regularizer expresses the inductive bias that sequence variables should have high mutual information even though the model might not see abundant observations for complex long-range dependency. We show how the “next sentence prediction (classification)" heuristic can be derived in a principled way from our mutual information estimation framework, and be further extended to maximize the mutual information of sequence variables. The proposed approach not only is effective at increasing the mutual information of segments under the learned model but more importantly, leads to a higher likelihood on holdout data, and improved generation quality. Code is releasedat https://github.com/BorealisAI/BMI.

Cite this Paper

BibTeX
@InProceedings{pmlr-v108-cao20a, title = {Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer}, author = {Cao, Yanshuai and Xu, Peng}, booktitle = {Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics}, pages = {3991--4001}, year = {2020}, editor = {Chiappa, Silvia and Calandra, Roberto}, volume = {108}, series = {Proceedings of Machine Learning Research}, month = {26--28 Aug}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v108/cao20a/cao20a.pdf}, url = {https://proceedings.mlr.press/v108/cao20a.html}, abstract = {In this work, we develop a novel regularizer to improve the learning of long-range dependency of sequence data. Applied on language modelling, our regularizer expresses the inductive bias that sequence variables should have high mutual information even though the model might not see abundant observations for complex long-range dependency. We show how the “next sentence prediction (classification)" heuristic can be derived in a principled way from our mutual information estimation framework, and be further extended to maximize the mutual information of sequence variables. The proposed approach not only is effective at increasing the mutual information of segments under the learned model but more importantly, leads to a higher likelihood on holdout data, and improved generation quality. Code is releasedat https://github.com/BorealisAI/BMI.} }
Endnote
%0 Conference Paper %T Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer %A Yanshuai Cao %A Peng Xu %B Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics %C Proceedings of Machine Learning Research %D 2020 %E Silvia Chiappa %E Roberto Calandra %F pmlr-v108-cao20a %I PMLR %P 3991--4001 %U https://proceedings.mlr.press/v108/cao20a.html %V 108 %X In this work, we develop a novel regularizer to improve the learning of long-range dependency of sequence data. Applied on language modelling, our regularizer expresses the inductive bias that sequence variables should have high mutual information even though the model might not see abundant observations for complex long-range dependency. We show how the “next sentence prediction (classification)" heuristic can be derived in a principled way from our mutual information estimation framework, and be further extended to maximize the mutual information of sequence variables. The proposed approach not only is effective at increasing the mutual information of segments under the learned model but more importantly, leads to a higher likelihood on holdout data, and improved generation quality. Code is releasedat https://github.com/BorealisAI/BMI.
APA
Cao, Y. & Xu, P.. (2020). Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer. Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, in Proceedings of Machine Learning Research 108:3991-4001 Available from https://proceedings.mlr.press/v108/cao20a.html.

Related Material