Offline Model-Based Reinforcement Learning for Tokamak Control

Ian Char, Joseph Abbate, Laszlo Bardoczi, Mark Boyer, Youngseog Chung, Rory Conlin, Keith Erickson, Viraj Mehta, Nathan Richner, Egemen Kolemen, Jeff Schneider
Proceedings of The 5th Annual Learning for Dynamics and Control Conference, PMLR 211:1357-1372, 2023.

Abstract

Control for tokamaks, the leading candidate technology for nuclear fusion, is an important pursuit since the realization of nuclear fusion as an energy source would result in virtually unlimited clean energy. However, control of these devices remains a challenging problem due to complex, non-linear dynamics. At the same time, there is promise in learning controllers for difficult problems thanks to recent algorithmic developments in reinforcement learning. Because every run (or shot) of the tokamak is extremely expensive, in this work, we investigated learning a controller from logged data before testing it on a tokamak. In particular, we used 18 years of data from the DIII-D device in order to learn a controller for the neutral beams that targets specified $\beta_N$ (normalized ratio of plasma pressure to magnetic pressure) and rotation quantities. This was done by using the data to first learn a dynamics model, and then by using this model as a simulator to generate experience to train a controller via reinforcement learning. During a control session on DIII-D, we tested both the ability for our dynamics model to design feedforward trajectories and the controller’s ability to do feedback control to achieve specified targets. This work marks some of the first steps in doing reinforcement learning for tokamak control through historical data alone.

Cite this Paper

BibTeX
@InProceedings{pmlr-v211-char23a, title = {Offline Model-Based Reinforcement Learning for Tokamak Control}, author = {Char, Ian and Abbate, Joseph and Bardoczi, Laszlo and Boyer, Mark and Chung, Youngseog and Conlin, Rory and Erickson, Keith and Mehta, Viraj and Richner, Nathan and Kolemen, Egemen and Schneider, Jeff}, booktitle = {Proceedings of The 5th Annual Learning for Dynamics and Control Conference}, pages = {1357--1372}, year = {2023}, editor = {Matni, Nikolai and Morari, Manfred and Pappas, George J.}, volume = {211}, series = {Proceedings of Machine Learning Research}, month = {15--16 Jun}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v211/char23a/char23a.pdf}, url = {https://proceedings.mlr.press/v211/char23a.html}, abstract = {Control for tokamaks, the leading candidate technology for nuclear fusion, is an important pursuit since the realization of nuclear fusion as an energy source would result in virtually unlimited clean energy. However, control of these devices remains a challenging problem due to complex, non-linear dynamics. At the same time, there is promise in learning controllers for difficult problems thanks to recent algorithmic developments in reinforcement learning. Because every run (or shot) of the tokamak is extremely expensive, in this work, we investigated learning a controller from logged data before testing it on a tokamak. In particular, we used 18 years of data from the DIII-D device in order to learn a controller for the neutral beams that targets specified $\beta_N$ (normalized ratio of plasma pressure to magnetic pressure) and rotation quantities. This was done by using the data to first learn a dynamics model, and then by using this model as a simulator to generate experience to train a controller via reinforcement learning. During a control session on DIII-D, we tested both the ability for our dynamics model to design feedforward trajectories and the controller’s ability to do feedback control to achieve specified targets. This work marks some of the first steps in doing reinforcement learning for tokamak control through historical data alone.} }
Endnote
%0 Conference Paper %T Offline Model-Based Reinforcement Learning for Tokamak Control %A Ian Char %A Joseph Abbate %A Laszlo Bardoczi %A Mark Boyer %A Youngseog Chung %A Rory Conlin %A Keith Erickson %A Viraj Mehta %A Nathan Richner %A Egemen Kolemen %A Jeff Schneider %B Proceedings of The 5th Annual Learning for Dynamics and Control Conference %C Proceedings of Machine Learning Research %D 2023 %E Nikolai Matni %E Manfred Morari %E George J. Pappas %F pmlr-v211-char23a %I PMLR %P 1357--1372 %U https://proceedings.mlr.press/v211/char23a.html %V 211 %X Control for tokamaks, the leading candidate technology for nuclear fusion, is an important pursuit since the realization of nuclear fusion as an energy source would result in virtually unlimited clean energy. However, control of these devices remains a challenging problem due to complex, non-linear dynamics. At the same time, there is promise in learning controllers for difficult problems thanks to recent algorithmic developments in reinforcement learning. Because every run (or shot) of the tokamak is extremely expensive, in this work, we investigated learning a controller from logged data before testing it on a tokamak. In particular, we used 18 years of data from the DIII-D device in order to learn a controller for the neutral beams that targets specified $\beta_N$ (normalized ratio of plasma pressure to magnetic pressure) and rotation quantities. This was done by using the data to first learn a dynamics model, and then by using this model as a simulator to generate experience to train a controller via reinforcement learning. During a control session on DIII-D, we tested both the ability for our dynamics model to design feedforward trajectories and the controller’s ability to do feedback control to achieve specified targets. This work marks some of the first steps in doing reinforcement learning for tokamak control through historical data alone.
APA
Char, I., Abbate, J., Bardoczi, L., Boyer, M., Chung, Y., Conlin, R., Erickson, K., Mehta, V., Richner, N., Kolemen, E. & Schneider, J.. (2023). Offline Model-Based Reinforcement Learning for Tokamak Control. Proceedings of The 5th Annual Learning for Dynamics and Control Conference, in Proceedings of Machine Learning Research 211:1357-1372 Available from https://proceedings.mlr.press/v211/char23a.html.

Related Material