Stable modular control via contraction theory for reinforcement learning

We propose a novel way to integrate control theoretical results with reinforcement learning (RL) for stability, robustness, and generalization: developing modular control architectures via contraction theory to simplify the complex problems. To guarantee control stability for RL, we leverage modularity to deconstruct the nonlinear stability problems into algebraically solvable ones, yielding linear constraints on the input gradients of control networks that can be as simple as switching the signs of network weights. This control architecture can be implemented in general RL frameworks without modifying the algorithms. This minimally invasive way allows arguably easy integration into hierarchical RL, and improves its performance. We realize the modularity by constructing an auxiliary space through coordinate transformation. Within the auxiliary space, system dynamics can be represented as hierarchical combinations of subsystems. These subsystems converge recursively following their hierarchies, provided stable self-feedbacks. We implement this modular control architecture in PPO and hierarchical RL, and demonstrate in simulation (i) the necessity of control stability for robustness and generalization and (ii) the effectiveness in improving hierarchical RL for manipulation learning.