Hindsight Learning for MDPs with Exogenous Inputs

Sean R. Sinclair, Felipe Vieira Frujeri, Ching-An Cheng, Luke Marshall, Hugo De Oliveira Barbalho, Jingling Li, Jennifer Neville, Ishai Menache, Adith Swaminathan
Proceedings of the 40th International Conference on Machine Learning, PMLR 202:31877-31914, 2023.

Abstract

Many resource management problems require sequential decision-making under uncertainty, where the only uncertainty affecting the decision outcomes are exogenous variables outside the control of the decision-maker. We model these problems as Exo-MDPs (Markov Decision Processes with Exogenous Inputs) and design a class of data-efficient algorithms for them termed Hindsight Learning (HL). Our HL algorithms achieve data efficiency by leveraging a key insight: having samples of the exogenous variables, past decisions can be revisited in hindsight to infer counterfactual consequences that can accelerate policy improvements. We compare HL against classic baselines in the multi-secretary and airline revenue management problems. We also scale our algorithms to a business-critical cloud resource management problem – allocating Virtual Machines (VMs) to physical machines, and simulate their performance with real datasets from a large public cloud provider. We find that HL algorithms outperform domain-specific heuristics, as well as state-of-the-art reinforcement learning methods.

Cite this Paper

BibTeX
@InProceedings{pmlr-v202-sinclair23a, title = {Hindsight Learning for {MDP}s with Exogenous Inputs}, author = {Sinclair, Sean R. and Vieira Frujeri, Felipe and Cheng, Ching-An and Marshall, Luke and Barbalho, Hugo De Oliveira and Li, Jingling and Neville, Jennifer and Menache, Ishai and Swaminathan, Adith}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {31877--31914}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, month = {23--29 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v202/sinclair23a/sinclair23a.pdf}, url = {https://proceedings.mlr.press/v202/sinclair23a.html}, abstract = {Many resource management problems require sequential decision-making under uncertainty, where the only uncertainty affecting the decision outcomes are exogenous variables outside the control of the decision-maker. We model these problems as Exo-MDPs (Markov Decision Processes with Exogenous Inputs) and design a class of data-efficient algorithms for them termed Hindsight Learning (HL). Our HL algorithms achieve data efficiency by leveraging a key insight: having samples of the exogenous variables, past decisions can be revisited in hindsight to infer counterfactual consequences that can accelerate policy improvements. We compare HL against classic baselines in the multi-secretary and airline revenue management problems. We also scale our algorithms to a business-critical cloud resource management problem – allocating Virtual Machines (VMs) to physical machines, and simulate their performance with real datasets from a large public cloud provider. We find that HL algorithms outperform domain-specific heuristics, as well as state-of-the-art reinforcement learning methods.} }
Endnote
%0 Conference Paper %T Hindsight Learning for MDPs with Exogenous Inputs %A Sean R. Sinclair %A Felipe Vieira Frujeri %A Ching-An Cheng %A Luke Marshall %A Hugo De Oliveira Barbalho %A Jingling Li %A Jennifer Neville %A Ishai Menache %A Adith Swaminathan %B Proceedings of the 40th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2023 %E Andreas Krause %E Emma Brunskill %E Kyunghyun Cho %E Barbara Engelhardt %E Sivan Sabato %E Jonathan Scarlett %F pmlr-v202-sinclair23a %I PMLR %P 31877--31914 %U https://proceedings.mlr.press/v202/sinclair23a.html %V 202 %X Many resource management problems require sequential decision-making under uncertainty, where the only uncertainty affecting the decision outcomes are exogenous variables outside the control of the decision-maker. We model these problems as Exo-MDPs (Markov Decision Processes with Exogenous Inputs) and design a class of data-efficient algorithms for them termed Hindsight Learning (HL). Our HL algorithms achieve data efficiency by leveraging a key insight: having samples of the exogenous variables, past decisions can be revisited in hindsight to infer counterfactual consequences that can accelerate policy improvements. We compare HL against classic baselines in the multi-secretary and airline revenue management problems. We also scale our algorithms to a business-critical cloud resource management problem – allocating Virtual Machines (VMs) to physical machines, and simulate their performance with real datasets from a large public cloud provider. We find that HL algorithms outperform domain-specific heuristics, as well as state-of-the-art reinforcement learning methods.
APA
Sinclair, S.R., Vieira Frujeri, F., Cheng, C., Marshall, L., Barbalho, H.D.O., Li, J., Neville, J., Menache, I. & Swaminathan, A.. (2023). Hindsight Learning for MDPs with Exogenous Inputs. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:31877-31914 Available from https://proceedings.mlr.press/v202/sinclair23a.html.

Related Material