Designing and evaluating an online reinforcement learning agent for physical exercise recommendations in N-of-1 trials

Dominik Meier, Ipek Ensari, Stefan Konigorski
Proceedings of the 3rd Machine Learning for Health Symposium, PMLR 225:340-352, 2023.

Abstract

Personalized adaptive interventions offer the opportunity to increase patient benefits, however, there are challenges in their planning and implementation. Once implemented, it is an important question whether personalized adaptive interventions are indeed clinically more effective compared to a fixed gold standard intervention. In this paper, we present an innovative N-of-1 trial study design testing whether implementing a personalized intervention by an online reinforcement learning agent is feasible and effective. Throughout, we use a new study on physical exercise recommendations to reduce pain in endometriosis for illustration. We describe the design of a contextual bandit recommendation agent and evaluate the agent in simulation studies. The results show that, first, implementing a personalized intervention by an online reinforcement learning agent is feasible. Second, such adaptive interventions have the potential to improve patients’ benefits even if only few observations are available. As one challenge, they add complexity to the design and implementation process. In order to quantify the expected benefit, data from previous interventional studies is required. We expect our approach to be transferable to other interventions and clinical interventions.

Cite this Paper

BibTeX
@InProceedings{pmlr-v225-meier23a, title = {Designing and evaluating an online reinforcement learning agent for physical exercise recommendations in N-of-1 trials}, author = {Meier, Dominik and Ensari, Ipek and Konigorski, Stefan}, booktitle = {Proceedings of the 3rd Machine Learning for Health Symposium}, pages = {340--352}, year = {2023}, editor = {Hegselmann, Stefan and Parziale, Antonio and Shanmugam, Divya and Tang, Shengpu and Asiedu, Mercy Nyamewaa and Chang, Serina and Hartvigsen, Tom and Singh, Harvineet}, volume = {225}, series = {Proceedings of Machine Learning Research}, month = {10 Dec}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v225/meier23a/meier23a.pdf}, url = {https://proceedings.mlr.press/v225/meier23a.html}, abstract = {Personalized adaptive interventions offer the opportunity to increase patient benefits, however, there are challenges in their planning and implementation. Once implemented, it is an important question whether personalized adaptive interventions are indeed clinically more effective compared to a fixed gold standard intervention. In this paper, we present an innovative N-of-1 trial study design testing whether implementing a personalized intervention by an online reinforcement learning agent is feasible and effective. Throughout, we use a new study on physical exercise recommendations to reduce pain in endometriosis for illustration. We describe the design of a contextual bandit recommendation agent and evaluate the agent in simulation studies. The results show that, first, implementing a personalized intervention by an online reinforcement learning agent is feasible. Second, such adaptive interventions have the potential to improve patients’ benefits even if only few observations are available. As one challenge, they add complexity to the design and implementation process. In order to quantify the expected benefit, data from previous interventional studies is required. We expect our approach to be transferable to other interventions and clinical interventions.} }
Endnote
%0 Conference Paper %T Designing and evaluating an online reinforcement learning agent for physical exercise recommendations in N-of-1 trials %A Dominik Meier %A Ipek Ensari %A Stefan Konigorski %B Proceedings of the 3rd Machine Learning for Health Symposium %C Proceedings of Machine Learning Research %D 2023 %E Stefan Hegselmann %E Antonio Parziale %E Divya Shanmugam %E Shengpu Tang %E Mercy Nyamewaa Asiedu %E Serina Chang %E Tom Hartvigsen %E Harvineet Singh %F pmlr-v225-meier23a %I PMLR %P 340--352 %U https://proceedings.mlr.press/v225/meier23a.html %V 225 %X Personalized adaptive interventions offer the opportunity to increase patient benefits, however, there are challenges in their planning and implementation. Once implemented, it is an important question whether personalized adaptive interventions are indeed clinically more effective compared to a fixed gold standard intervention. In this paper, we present an innovative N-of-1 trial study design testing whether implementing a personalized intervention by an online reinforcement learning agent is feasible and effective. Throughout, we use a new study on physical exercise recommendations to reduce pain in endometriosis for illustration. We describe the design of a contextual bandit recommendation agent and evaluate the agent in simulation studies. The results show that, first, implementing a personalized intervention by an online reinforcement learning agent is feasible. Second, such adaptive interventions have the potential to improve patients’ benefits even if only few observations are available. As one challenge, they add complexity to the design and implementation process. In order to quantify the expected benefit, data from previous interventional studies is required. We expect our approach to be transferable to other interventions and clinical interventions.
APA
Meier, D., Ensari, I. & Konigorski, S.. (2023). Designing and evaluating an online reinforcement learning agent for physical exercise recommendations in N-of-1 trials. Proceedings of the 3rd Machine Learning for Health Symposium, in Proceedings of Machine Learning Research 225:340-352 Available from https://proceedings.mlr.press/v225/meier23a.html.

Related Material