Proximal Policy Optimization Suffices for On-Policy Reinforcement Learning

Rayane Bouftini, Mohammed Benzaouia
Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, PMLR 337:640-657, 2026.

Abstract

Proximal Policy Optimization ({PPO}) has been applied to a wide range of applications, from robotics to large language models ({LLMs}). Yet, despite its success, it still suffers from unstable updates, performance collapse, and extreme sensitivity to hyperparameters. Many alternatives have been proposed to alleviate these issues, often demonstrating empirically stronger performance on standard reinforcement learning ({RL}) benchmarks. In this work, we observe that {PPO}’s training instability fundamentally stems from the brittle inter-coupling between its surrogate optimization hyperparameters which jointly dictate the effective size of policy updates. This complex coupling makes tuning quite difficult and, in practice, often leaves {PPO} under-performing. To mitigate this, we introduce a criterion on trajectories generated by behavior and target policies which can be used as an early stopping rule. This formulation effectively decouples the optimization dynamics, allowing the algorithm to adaptively scale its updates using a single threshold parameter. We additionally find a scaling law between this threshold parameter and the number of trajectory batches collected, further reducing the tuning burden. Using this simple criterion, we evaluate {PPO} on continuous control tasks from the {DM} Control Suite, outperforming prior policy gradient methods and revealing that many reported improvements were largely artifacts of under-performing {PPO} baselines.

Cite this Paper

BibTeX
@InProceedings{pmlr-v337-bouftini26a, title = {Proximal Policy Optimization Suffices for On-Policy Reinforcement Learning}, author = {Bouftini, Rayane and Benzaouia, Mohammed}, booktitle = {Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence}, pages = {640--657}, year = {2026}, editor = {Perković, Emilija and Malinsky, Daniel}, volume = {337}, series = {Proceedings of Machine Learning Research}, month = {17--21 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v337/main/assets/bouftini26a/bouftini26a.pdf}, url = {https://proceedings.mlr.press/v337/bouftini26a.html}, abstract = {Proximal Policy Optimization ({PPO}) has been applied to a wide range of applications, from robotics to large language models ({LLMs}). Yet, despite its success, it still suffers from unstable updates, performance collapse, and extreme sensitivity to hyperparameters. Many alternatives have been proposed to alleviate these issues, often demonstrating empirically stronger performance on standard reinforcement learning ({RL}) benchmarks. In this work, we observe that {PPO}’s training instability fundamentally stems from the brittle inter-coupling between its surrogate optimization hyperparameters which jointly dictate the effective size of policy updates. This complex coupling makes tuning quite difficult and, in practice, often leaves {PPO} under-performing. To mitigate this, we introduce a criterion on trajectories generated by behavior and target policies which can be used as an early stopping rule. This formulation effectively decouples the optimization dynamics, allowing the algorithm to adaptively scale its updates using a single threshold parameter. We additionally find a scaling law between this threshold parameter and the number of trajectory batches collected, further reducing the tuning burden. Using this simple criterion, we evaluate {PPO} on continuous control tasks from the {DM} Control Suite, outperforming prior policy gradient methods and revealing that many reported improvements were largely artifacts of under-performing {PPO} baselines.} }
Endnote
%0 Conference Paper %T Proximal Policy Optimization Suffices for On-Policy Reinforcement Learning %A Rayane Bouftini %A Mohammed Benzaouia %B Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2026 %E Emilija Perković %E Daniel Malinsky %F pmlr-v337-bouftini26a %I PMLR %P 640--657 %U https://proceedings.mlr.press/v337/bouftini26a.html %V 337 %X Proximal Policy Optimization ({PPO}) has been applied to a wide range of applications, from robotics to large language models ({LLMs}). Yet, despite its success, it still suffers from unstable updates, performance collapse, and extreme sensitivity to hyperparameters. Many alternatives have been proposed to alleviate these issues, often demonstrating empirically stronger performance on standard reinforcement learning ({RL}) benchmarks. In this work, we observe that {PPO}’s training instability fundamentally stems from the brittle inter-coupling between its surrogate optimization hyperparameters which jointly dictate the effective size of policy updates. This complex coupling makes tuning quite difficult and, in practice, often leaves {PPO} under-performing. To mitigate this, we introduce a criterion on trajectories generated by behavior and target policies which can be used as an early stopping rule. This formulation effectively decouples the optimization dynamics, allowing the algorithm to adaptively scale its updates using a single threshold parameter. We additionally find a scaling law between this threshold parameter and the number of trajectory batches collected, further reducing the tuning burden. Using this simple criterion, we evaluate {PPO} on continuous control tasks from the {DM} Control Suite, outperforming prior policy gradient methods and revealing that many reported improvements were largely artifacts of under-performing {PPO} baselines.
APA
Bouftini, R. & Benzaouia, M.. (2026). Proximal Policy Optimization Suffices for On-Policy Reinforcement Learning. Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 337:640-657 Available from https://proceedings.mlr.press/v337/bouftini26a.html.

Related Material