Lagrangian Method for Q-Function Learning (with Applications to Machine Translation)

Huang Bojun
Proceedings of the 39th International Conference on Machine Learning, PMLR 162:2129-2159, 2022.

Abstract

This paper discusses a new approach to the fundamental problem of learning optimal Q-functions. In this approach, optimal Q-functions are formulated as saddle points of a nonlinear Lagrangian function derived from the classic Bellman optimality equation. The paper shows that the Lagrangian enjoys strong duality, in spite of its nonlinearity, which paves the way to a general Lagrangian method to Q-function learning. As a demonstration, the paper develops an imitation learning algorithm based on the duality theory, and applies the algorithm to a state-of-the-art machine translation benchmark. The paper then turns to demonstrate a symmetry breaking phenomenon regarding the optimality of the Lagrangian saddle points, which justifies a largely overlooked direction in developing the Lagrangian method.

Cite this Paper

BibTeX
@InProceedings{pmlr-v162-bojun22a, title = {Lagrangian Method for Q-Function Learning (with Applications to Machine Translation)}, author = {Bojun, Huang}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {2129--2159}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, month = {17--23 Jul}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v162/bojun22a/bojun22a.pdf}, url = {https://proceedings.mlr.press/v162/bojun22a.html}, abstract = {This paper discusses a new approach to the fundamental problem of learning optimal Q-functions. In this approach, optimal Q-functions are formulated as saddle points of a nonlinear Lagrangian function derived from the classic Bellman optimality equation. The paper shows that the Lagrangian enjoys strong duality, in spite of its nonlinearity, which paves the way to a general Lagrangian method to Q-function learning. As a demonstration, the paper develops an imitation learning algorithm based on the duality theory, and applies the algorithm to a state-of-the-art machine translation benchmark. The paper then turns to demonstrate a symmetry breaking phenomenon regarding the optimality of the Lagrangian saddle points, which justifies a largely overlooked direction in developing the Lagrangian method.} }
Endnote
%0 Conference Paper %T Lagrangian Method for Q-Function Learning (with Applications to Machine Translation) %A Huang Bojun %B Proceedings of the 39th International Conference on Machine Learning %C Proceedings of Machine Learning Research %D 2022 %E Kamalika Chaudhuri %E Stefanie Jegelka %E Le Song %E Csaba Szepesvari %E Gang Niu %E Sivan Sabato %F pmlr-v162-bojun22a %I PMLR %P 2129--2159 %U https://proceedings.mlr.press/v162/bojun22a.html %V 162 %X This paper discusses a new approach to the fundamental problem of learning optimal Q-functions. In this approach, optimal Q-functions are formulated as saddle points of a nonlinear Lagrangian function derived from the classic Bellman optimality equation. The paper shows that the Lagrangian enjoys strong duality, in spite of its nonlinearity, which paves the way to a general Lagrangian method to Q-function learning. As a demonstration, the paper develops an imitation learning algorithm based on the duality theory, and applies the algorithm to a state-of-the-art machine translation benchmark. The paper then turns to demonstrate a symmetry breaking phenomenon regarding the optimality of the Lagrangian saddle points, which justifies a largely overlooked direction in developing the Lagrangian method.
APA
Bojun, H.. (2022). Lagrangian Method for Q-Function Learning (with Applications to Machine Translation). Proceedings of the 39th International Conference on Machine Learning, in Proceedings of Machine Learning Research 162:2129-2159 Available from https://proceedings.mlr.press/v162/bojun22a.html.

Related Material