The Surprising Difficulty of Search in Model-Based Reinforcement Learning

Wei-Di Chang, Mikael Henaff, Brandon Amos, Gregory Dudek, Scott Fujimoto
Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:12685-12717, 2026.

Abstract

This paper investigates search in model-based reinforcement learning (RL). Conventional wisdom holds that long-term predictions and compounding errors are the primary obstacles for model-based RL. We challenge this view, showing that search is not a drop-in replacement for a learned policy. Surprisingly, we find that search can harm performance even when the model is highly accurate. Instead, we show that mitigating overestimation bias matters more than improving model or value function accuracy. Building on this insight, we identify that taking the minimum over an ensemble of value functions effectively addresses this bias and enables effective search, achieving state-of-the-art performance across multiple popular benchmark domains. Code can be found at https://github.com/facebookresearch/MRSQ.

Cite this Paper

BibTeX
@InProceedings{pmlr-v306-chang26a, title = {The Surprising Difficulty of Search in Model-Based Reinforcement Learning}, author = {Chang, Wei-Di and Henaff, Mikael and Amos, Brandon and Dudek, Gregory and Fujimoto, Scott}, booktitle = {Proceedings of the 43rd International Conference on Machine Learning}, pages = {12685--12717}, year = {2026}, editor = {Zhang, Tong and Dudik, Miroslav and Jaggi, Martin and Agarwal, Alekh and Li, Sharon and Schuurmans, Dale and Zhu, Jerry and Berkenkamp, Felix and Dong, Hanze and Bietti, Alberto}, volume = {306}, series = {Proceedings of Machine Learning Research}, month = {06--11 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v306/main/assets/chang26a/chang26a.pdf}, url = {https://proceedings.mlr.press/v306/chang26a.html}, abstract = {This paper investigates search in model-based reinforcement learning (RL). Conventional wisdom holds that long-term predictions and compounding errors are the primary obstacles for model-based RL. We challenge this view, showing that search is not a drop-in replacement for a learned policy. Surprisingly, we find that search can harm performance even when the model is highly accurate. Instead, we show that mitigating overestimation bias matters more than improving model or value function accuracy. Building on this insight, we identify that taking the minimum over an ensemble of value functions effectively addresses this bias and enables effective search, achieving state-of-the-art performance across multiple popular benchmark domains. Code can be found at https://github.com/facebookresearch/MRSQ.} }
Endnote
%0 Conference Paper %T The Surprising Difficulty of Search in Model-Based Reinforcement Learning %A Wei-Di Chang %A Mikael Henaff %A Brandon Amos %A Gregory Dudek %A Scott Fujimoto %B Proceedings of the 43rd International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2026 %E Tong Zhang %E Miroslav Dudik %E Martin Jaggi %E Alekh Agarwal %E Sharon Li %E Dale Schuurmans %E Jerry Zhu %E Felix Berkenkamp %E Hanze Dong %E Alberto Bietti %F pmlr-v306-chang26a %I PMLR %P 12685--12717 %U https://proceedings.mlr.press/v306/chang26a.html %V 306 %X This paper investigates search in model-based reinforcement learning (RL). Conventional wisdom holds that long-term predictions and compounding errors are the primary obstacles for model-based RL. We challenge this view, showing that search is not a drop-in replacement for a learned policy. Surprisingly, we find that search can harm performance even when the model is highly accurate. Instead, we show that mitigating overestimation bias matters more than improving model or value function accuracy. Building on this insight, we identify that taking the minimum over an ensemble of value functions effectively addresses this bias and enables effective search, achieving state-of-the-art performance across multiple popular benchmark domains. Code can be found at https://github.com/facebookresearch/MRSQ.
APA
Chang, W., Henaff, M., Amos, B., Dudek, G. & Fujimoto, S.. (2026). The Surprising Difficulty of Search in Model-Based Reinforcement Learning. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 306:12685-12717 Available from https://proceedings.mlr.press/v306/chang26a.html.

Related Material