PAC-Bayesian Soft Actor-Critic Learning

Bahareh Tasdighi, Abdullah Akgül, Manuel Haussmann, Kenny Kazimirzak Brink, Melih Kandemir
Proceedings of the 6th Symposium on Advances in Approximate Bayesian Inference, PMLR 253:127-145, 2024.

Abstract

Actor-critic algorithms address the dual goals of reinforcement learning (RL), policy evaluation and improvement via two separate function approximators. The practicality of this approach comes at the expense of training instability, caused mainly by the destructive effect of the approximation errors of the critic on the actor. We tackle this bottleneck by employing an existing Probably Approximately Correct (PAC) Bayesian bound for the first time as the critic training objective of the Soft Actor-Critic (SAC) algorithm. We further demonstrate that online learning performance improves significantly when a stochastic actor explores multiple futures by critic-guided random search. We observe our resulting algorithm to compare favorably against the state-of-the-art SAC implementation on multiple classical control and locomotion tasks in terms of both sample efficiency and regret.

Cite this Paper

BibTeX
@InProceedings{pmlr-v253-tasdighi24a, title = {PAC-Bayesian Soft Actor-Critic Learning}, author = {Tasdighi, Bahareh and Akg{\"u}l, Abdullah and Haussmann, Manuel and Brink, Kenny Kazimirzak and Kandemir, Melih}, booktitle = {Proceedings of the 6th Symposium on Advances in Approximate Bayesian Inference}, pages = {127--145}, year = {2024}, editor = {Antorán, Javier and Naesseth, Christian A.}, volume = {253}, series = {Proceedings of Machine Learning Research}, month = {21 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v253/main/assets/tasdighi24a/tasdighi24a.pdf}, url = {https://proceedings.mlr.press/v253/tasdighi24a.html}, abstract = {Actor-critic algorithms address the dual goals of reinforcement learning (RL), policy evaluation and improvement via two separate function approximators. The practicality of this approach comes at the expense of training instability, caused mainly by the destructive effect of the approximation errors of the critic on the actor. We tackle this bottleneck by employing an existing Probably Approximately Correct (PAC) Bayesian bound for the first time as the critic training objective of the Soft Actor-Critic (SAC) algorithm. We further demonstrate that online learning performance improves significantly when a stochastic actor explores multiple futures by critic-guided random search. We observe our resulting algorithm to compare favorably against the state-of-the-art SAC implementation on multiple classical control and locomotion tasks in terms of both sample efficiency and regret.} }
Endnote
%0 Conference Paper %T PAC-Bayesian Soft Actor-Critic Learning %A Bahareh Tasdighi %A Abdullah Akgül %A Manuel Haussmann %A Kenny Kazimirzak Brink %A Melih Kandemir %B Proceedings of the 6th Symposium on Advances in Approximate Bayesian Inference %C Proceedings of Machine Learning Research %D 2024 %E Javier Antorán %E Christian A. Naesseth %F pmlr-v253-tasdighi24a %I PMLR %P 127--145 %U https://proceedings.mlr.press/v253/tasdighi24a.html %V 253 %X Actor-critic algorithms address the dual goals of reinforcement learning (RL), policy evaluation and improvement via two separate function approximators. The practicality of this approach comes at the expense of training instability, caused mainly by the destructive effect of the approximation errors of the critic on the actor. We tackle this bottleneck by employing an existing Probably Approximately Correct (PAC) Bayesian bound for the first time as the critic training objective of the Soft Actor-Critic (SAC) algorithm. We further demonstrate that online learning performance improves significantly when a stochastic actor explores multiple futures by critic-guided random search. We observe our resulting algorithm to compare favorably against the state-of-the-art SAC implementation on multiple classical control and locomotion tasks in terms of both sample efficiency and regret.
APA
Tasdighi, B., Akgül, A., Haussmann, M., Brink, K.K. & Kandemir, M.. (2024). PAC-Bayesian Soft Actor-Critic Learning. Proceedings of the 6th Symposium on Advances in Approximate Bayesian Inference, in Proceedings of Machine Learning Research 253:127-145 Available from https://proceedings.mlr.press/v253/tasdighi24a.html.

Related Material