Policy Learning for Balancing Short-Term and Long-Term Rewards

Peng Wu, Ziyu Shen, Feng Xie, Wang Zhongyao, Chunchen Liu, Yan Zeng
Proceedings of the 41st International Conference on Machine Learning, PMLR 235:53817-53846, 2024.

Abstract

Empirical researchers and decision-makers spanning various domains frequently seek profound insights into the long-term impacts of interventions. While the significance of long-term outcomes is undeniable, an overemphasis on them may inadvertently overshadow short-term gains. Motivated by this, this paper formalizes a new framework for learning the optimal policy that effectively balances both long-term and short-term rewards, where some long-term outcomes are allowed to be missing. In particular, we first present the identifiability of both rewards under mild assumptions. Next, we deduce the semiparametric efficiency bounds, along with the consistency and asymptotic normality of their estimators. We also reveal that short-term outcomes, if associated, contribute to improving the estimator of the long-term reward. Based on the proposed estimators, we develop a principled policy learning approach and further derive the convergence rates of regret and estimation errors associated with the learned policy. Extensive experiments are conducted to validate the effectiveness of the proposed method, demonstrating its practical applicability.

Cite this Paper

BibTeX
@InProceedings{pmlr-v235-wu24x, title = {Policy Learning for Balancing Short-Term and Long-Term Rewards}, author = {Wu, Peng and Shen, Ziyu and Xie, Feng and Zhongyao, Wang and Liu, Chunchen and Zeng, Yan}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {53817--53846}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/wu24x/wu24x.pdf}, url = {https://proceedings.mlr.press/v235/wu24x.html}, abstract = {Empirical researchers and decision-makers spanning various domains frequently seek profound insights into the long-term impacts of interventions. While the significance of long-term outcomes is undeniable, an overemphasis on them may inadvertently overshadow short-term gains. Motivated by this, this paper formalizes a new framework for learning the optimal policy that effectively balances both long-term and short-term rewards, where some long-term outcomes are allowed to be missing. In particular, we first present the identifiability of both rewards under mild assumptions. Next, we deduce the semiparametric efficiency bounds, along with the consistency and asymptotic normality of their estimators. We also reveal that short-term outcomes, if associated, contribute to improving the estimator of the long-term reward. Based on the proposed estimators, we develop a principled policy learning approach and further derive the convergence rates of regret and estimation errors associated with the learned policy. Extensive experiments are conducted to validate the effectiveness of the proposed method, demonstrating its practical applicability.} }
Endnote
%0 Conference Paper %T Policy Learning for Balancing Short-Term and Long-Term Rewards %A Peng Wu %A Ziyu Shen %A Feng Xie %A Wang Zhongyao %A Chunchen Liu %A Yan Zeng %B Proceedings of the 41st International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2024 %E Ruslan Salakhutdinov %E Zico Kolter %E Katherine Heller %E Adrian Weller %E Nuria Oliver %E Jonathan Scarlett %E Felix Berkenkamp %F pmlr-v235-wu24x %I PMLR %P 53817--53846 %U https://proceedings.mlr.press/v235/wu24x.html %V 235 %X Empirical researchers and decision-makers spanning various domains frequently seek profound insights into the long-term impacts of interventions. While the significance of long-term outcomes is undeniable, an overemphasis on them may inadvertently overshadow short-term gains. Motivated by this, this paper formalizes a new framework for learning the optimal policy that effectively balances both long-term and short-term rewards, where some long-term outcomes are allowed to be missing. In particular, we first present the identifiability of both rewards under mild assumptions. Next, we deduce the semiparametric efficiency bounds, along with the consistency and asymptotic normality of their estimators. We also reveal that short-term outcomes, if associated, contribute to improving the estimator of the long-term reward. Based on the proposed estimators, we develop a principled policy learning approach and further derive the convergence rates of regret and estimation errors associated with the learned policy. Extensive experiments are conducted to validate the effectiveness of the proposed method, demonstrating its practical applicability.
APA
Wu, P., Shen, Z., Xie, F., Zhongyao, W., Liu, C. & Zeng, Y.. (2024). Policy Learning for Balancing Short-Term and Long-Term Rewards. Proceedings of the 41st International Conference on Machine Learning, in Proceedings of Machine Learning Research 235:53817-53846 Available from https://proceedings.mlr.press/v235/wu24x.html.

Related Material