Revisiting TD Target Aggregation under Uncertainty in Q-Learning

Lipeng Zu, Xiaonan Zhang
Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, PMLR 337:8330-8348, 2026.

Abstract

Deep Q-Networks (DQNs) learn value functions through bootstrapped temporal-difference updates, where future returns are approximated using a greedy maximization over next-state action values. While effective, this aggregation rule is inherently sensitive to estimation noise: when Q-values are uncertain, the maximization operator deterministically favors the largest estimate, regardless of its reliability, leading to amplified errors through bootstrapping. In this work, we propose the **S**uccessor Rollout **A**ggregation **D**eep **Q**-Network (SADQ), a simple modification to Q-learning that regularizes how the {TD} target is formed. SADQ uses one-step rollout predictions from a learned dynamics model to guide the comparison among candidate next-state actions, introducing additional structure into the aggregation step without altering the underlying learning framework. The resulting mixed {Bellman} update attenuates unreliable maxima while preserving the standard fixed point under diminishing model error. We provide theoretical analysis showing that SADQ reduces bootstrap-induced overestimation in a pointwise manner. Empirically, SADQ consistently improves training stability across classical control tasks, real-world vector-based environments, and Atari benchmarks when compared to strong DQN variants.

Cite this Paper

BibTeX
@InProceedings{pmlr-v337-zu26a, title = {Revisiting {TD} Target Aggregation under Uncertainty in {Q-Learning}}, author = {Zu, Lipeng and Zhang, Xiaonan}, booktitle = {Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence}, pages = {8330--8348}, year = {2026}, editor = {Perković, Emilija and Malinsky, Daniel}, volume = {337}, series = {Proceedings of Machine Learning Research}, month = {17--21 Aug}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v337/main/assets/zu26a/zu26a.pdf}, url = {https://proceedings.mlr.press/v337/zu26a.html}, abstract = {Deep Q-Networks (DQNs) learn value functions through bootstrapped temporal-difference updates, where future returns are approximated using a greedy maximization over next-state action values. While effective, this aggregation rule is inherently sensitive to estimation noise: when Q-values are uncertain, the maximization operator deterministically favors the largest estimate, regardless of its reliability, leading to amplified errors through bootstrapping. In this work, we propose the **S**uccessor Rollout **A**ggregation **D**eep **Q**-Network (SADQ), a simple modification to Q-learning that regularizes how the {TD} target is formed. SADQ uses one-step rollout predictions from a learned dynamics model to guide the comparison among candidate next-state actions, introducing additional structure into the aggregation step without altering the underlying learning framework. The resulting mixed {Bellman} update attenuates unreliable maxima while preserving the standard fixed point under diminishing model error. We provide theoretical analysis showing that SADQ reduces bootstrap-induced overestimation in a pointwise manner. Empirically, SADQ consistently improves training stability across classical control tasks, real-world vector-based environments, and Atari benchmarks when compared to strong DQN variants.} }
Endnote
%0 Conference Paper %T Revisiting TD Target Aggregation under Uncertainty in Q-Learning %A Lipeng Zu %A Xiaonan Zhang %B Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence %C Proceedings of Machine Learning Research %D 2026 %E Emilija Perković %E Daniel Malinsky %F pmlr-v337-zu26a %I PMLR %P 8330--8348 %U https://proceedings.mlr.press/v337/zu26a.html %V 337 %X Deep Q-Networks (DQNs) learn value functions through bootstrapped temporal-difference updates, where future returns are approximated using a greedy maximization over next-state action values. While effective, this aggregation rule is inherently sensitive to estimation noise: when Q-values are uncertain, the maximization operator deterministically favors the largest estimate, regardless of its reliability, leading to amplified errors through bootstrapping. In this work, we propose the **S**uccessor Rollout **A**ggregation **D**eep **Q**-Network (SADQ), a simple modification to Q-learning that regularizes how the {TD} target is formed. SADQ uses one-step rollout predictions from a learned dynamics model to guide the comparison among candidate next-state actions, introducing additional structure into the aggregation step without altering the underlying learning framework. The resulting mixed {Bellman} update attenuates unreliable maxima while preserving the standard fixed point under diminishing model error. We provide theoretical analysis showing that SADQ reduces bootstrap-induced overestimation in a pointwise manner. Empirically, SADQ consistently improves training stability across classical control tasks, real-world vector-based environments, and Atari benchmarks when compared to strong DQN variants.
APA
Zu, L. & Zhang, X.. (2026). Revisiting TD Target Aggregation under Uncertainty in Q-Learning. Proceedings of the 42nd Conference on Uncertainty in Artificial Intelligence, in Proceedings of Machine Learning Research 337:8330-8348 Available from https://proceedings.mlr.press/v337/zu26a.html.

Related Material