Entropy Minimization In Emergent Languages

Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt, Marco Baroni
Proceedings of the 37th International Conference on Machine Learning, PMLR 119:5220-5230, 2020.

Abstract

There is growing interest in studying the languages that emerge when neural agents are jointly trained to solve tasks requiring communication through a discrete channel. We investigate here the information-theoretic complexity of such languages, focusing on the basic two-agent, one-exchange setup. We find that, under common training procedures, the emergent languages are subject to an entropy minimization pressure that has also been detected in human language, whereby the mutual information between the communicating agent’s inputs and the messages is minimized, within the range afforded by the need for successful communication. That is, emergent languages are (nearly) as simple as the task they are developed for allow them to be. This pressure is amplified as we increase communication channel discreteness. Further, we observe that stronger discrete-channel-driven entropy minimization leads to representations with increased robustness to overfitting and adversarial attacks. We conclude by discussing the implications of our findings for the study of natural and artificial communication systems.

Cite this Paper

BibTeX
@InProceedings{pmlr-v119-kharitonov20a, title = {Entropy Minimization In Emergent Languages}, author = {Kharitonov, Eugene and Chaabouni, Rahma and Bouchacourt, Diane and Baroni, Marco}, booktitle = {Proceedings of the 37th International Conference on Machine Learning}, pages = {5220--5230}, year = {2020}, editor = {III, Hal Daumé and Singh, Aarti}, volume = {119}, series = {Proceedings of Machine Learning Research}, month = {13--18 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v119/kharitonov20a/kharitonov20a.pdf}, url = {https://proceedings.mlr.press/v119/kharitonov20a.html}, abstract = {There is growing interest in studying the languages that emerge when neural agents are jointly trained to solve tasks requiring communication through a discrete channel. We investigate here the information-theoretic complexity of such languages, focusing on the basic two-agent, one-exchange setup. We find that, under common training procedures, the emergent languages are subject to an entropy minimization pressure that has also been detected in human language, whereby the mutual information between the communicating agent’s inputs and the messages is minimized, within the range afforded by the need for successful communication. That is, emergent languages are (nearly) as simple as the task they are developed for allow them to be. This pressure is amplified as we increase communication channel discreteness. Further, we observe that stronger discrete-channel-driven entropy minimization leads to representations with increased robustness to overfitting and adversarial attacks. We conclude by discussing the implications of our findings for the study of natural and artificial communication systems.} }
Endnote
%0 Conference Paper %T Entropy Minimization In Emergent Languages %A Eugene Kharitonov %A Rahma Chaabouni %A Diane Bouchacourt %A Marco Baroni %B Proceedings of the 37th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2020 %E Hal Daumé III %E Aarti Singh %F pmlr-v119-kharitonov20a %I PMLR %P 5220--5230 %U https://proceedings.mlr.press/v119/kharitonov20a.html %V 119 %X There is growing interest in studying the languages that emerge when neural agents are jointly trained to solve tasks requiring communication through a discrete channel. We investigate here the information-theoretic complexity of such languages, focusing on the basic two-agent, one-exchange setup. We find that, under common training procedures, the emergent languages are subject to an entropy minimization pressure that has also been detected in human language, whereby the mutual information between the communicating agent’s inputs and the messages is minimized, within the range afforded by the need for successful communication. That is, emergent languages are (nearly) as simple as the task they are developed for allow them to be. This pressure is amplified as we increase communication channel discreteness. Further, we observe that stronger discrete-channel-driven entropy minimization leads to representations with increased robustness to overfitting and adversarial attacks. We conclude by discussing the implications of our findings for the study of natural and artificial communication systems.
APA
Kharitonov, E., Chaabouni, R., Bouchacourt, D. & Baroni, M.. (2020). Entropy Minimization In Emergent Languages. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:5220-5230 Available from https://proceedings.mlr.press/v119/kharitonov20a.html.

Related Material