Emergence of Adaptive Circadian Rhythms in Deep Reinforcement Learning

Aqeel Labash, Florian Stelzer, Daniel Majoral, Raul Vicente Zafra
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:18153-18170, 2023.

Abstract

Adapting to regularities of the environment is critical for biological organisms to anticipate events and plan. A prominent example is the circadian rhythm corresponding to the internalization by organisms of the $24$-hour period of the Earth’s rotation. In this work, we study the emergence of circadian-like rhythms in deep reinforcement learning agents. In particular, we deployed agents in an environment with a reliable periodic variation while solving a foraging task. We systematically characterize the agent’s behavior during learning and demonstrate the emergence of a rhythm that is endogenous and entrainable. Interestingly, the internal rhythm adapts to shifts in the phase of the environmental signal without any re-training. Furthermore, we show via bifurcation and phase response curve analyses how artificial neurons develop dynamics to support the internalization of the environmental rhythm. From a dynamical systems view, we demonstrate that the adaptation proceeds by the emergence of a stable periodic orbit in the neuron dynamics with a phase response that allows an optimal phase synchronisation between the agent’s dynamics and the environmental rhythm.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-labash23a, title = {Emergence of Adaptive Circadian Rhythms in Deep Reinforcement Learning}, author = {Labash, Aqeel and Stelzer, Florian and Majoral, Daniel and Vicente Zafra, Raul}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {18153--18170}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/labash23a/labash23a.pdf}, url = {https://proceedings.mlr.press/v202/labash23a.html}, abstract = {Adapting to regularities of the environment is critical for biological organisms to anticipate events and plan. A prominent example is the circadian rhythm corresponding to the internalization by organisms of the $24$-hour period of the Earth’s rotation. In this work, we study the emergence of circadian-like rhythms in deep reinforcement learning agents. In particular, we deployed agents in an environment with a reliable periodic variation while solving a foraging task. We systematically characterize the agent’s behavior during learning and demonstrate the emergence of a rhythm that is endogenous and entrainable. Interestingly, the internal rhythm adapts to shifts in the phase of the environmental signal without any re-training. Furthermore, we show via bifurcation and phase response curve analyses how artificial neurons develop dynamics to support the internalization of the environmental rhythm. From a dynamical systems view, we demonstrate that the adaptation proceeds by the emergence of a stable periodic orbit in the neuron dynamics with a phase response that allows an optimal phase synchronisation between the agent’s dynamics and the environmental rhythm.} }
Endnote
%0 Conference Paper %T Emergence of Adaptive Circadian Rhythms in Deep Reinforcement Learning %A Aqeel Labash %A Florian Stelzer %A Daniel Majoral %A Raul Vicente Zafra %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-labash23a %I PMLR %P 18153--18170 %U https://proceedings.mlr.press/v202/labash23a.html %V 202 %X Adapting to regularities of the environment is critical for biological organisms to anticipate events and plan. A prominent example is the circadian rhythm corresponding to the internalization by organisms of the $24$-hour period of the Earth’s rotation. In this work, we study the emergence of circadian-like rhythms in deep reinforcement learning agents. In particular, we deployed agents in an environment with a reliable periodic variation while solving a foraging task. We systematically characterize the agent’s behavior during learning and demonstrate the emergence of a rhythm that is endogenous and entrainable. Interestingly, the internal rhythm adapts to shifts in the phase of the environmental signal without any re-training. Furthermore, we show via bifurcation and phase response curve analyses how artificial neurons develop dynamics to support the internalization of the environmental rhythm. From a dynamical systems view, we demonstrate that the adaptation proceeds by the emergence of a stable periodic orbit in the neuron dynamics with a phase response that allows an optimal phase synchronisation between the agent’s dynamics and the environmental rhythm.
APA
Labash, A., Stelzer, F., Majoral, D. & Vicente Zafra, R.. (2023). Emergence of Adaptive Circadian Rhythms in Deep Reinforcement Learning. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:18153-18170 Available from https://proceedings.mlr.press/v202/labash23a.html.

Related Material