[edit]

# On the nonsmooth geometry and neural approximation of the optimal value function of infinite-horizon pendulum swing-up

*Proceedings of the 6th Annual Learning for Dynamics & Control Conference*, PMLR 242:654-666, 2024.

#### Abstract

We revisit the inverted pendulum problem with the goal of understanding and computing the true optimal value function. We start with an observation that the true optimal value function must be nonsmooth (i.e., not globally C1) due to symmetry of the problem. We then give a result that can certify the optimality of a candidate piece-wise C1 value function. Further, for a candidate value function obtained via numerical approximation, we provide a bound of suboptimality based on its Hamilton-Jacobi-Bellman (HJB) equation residuals. Inspired by Holzhüter (2004), we then design an algorithm that solves backwards the Pontryagin’s minimum principle (PMP) ODE from terminal conditions provided by the locally optimal LQR value function. This numerical procedure leads to a piece-wise C1 value function whose nonsmooth region contains periodic spiral lines and smooth regions attain HJB residuals about $10^{-4}$, hence certiﬁed to be the optimal value function up to minor numerical inaccuracies. This optimal value function checks the power of optimality: (i) it sits above a polynomial lower bound; (ii) its induced controller globally swings up and stabilizes the pendulum, and (iii) attains lower trajectory cost than baseline methods such as energy shaping, model predictive control (MPC), and proximal policy optimization (with MPC attaining almost the same cost). We conclude by distilling the optimal value function into a simple neural network.