A Covariance Matrix Adaptation Evolution Strategy for Direct Policy Search in Reproducing Kernel Hilbert Space
[edit]
Proceedings of the Ninth Asian Conference on Machine Learning, PMLR 77:606621, 2017.
Abstract
The covariance matrix adaptation evolution strategy (CMAES) is an efficient derivativefree optimization algorithm. It optimizes a blackbox objective function over a well defined parameter space. In some problems, such parameter spaces are defined using function approximation in which feature functions are manually defined. Therefore, the performance of those techniques strongly depends on the quality of chosen features. Hence, enabling CMAES to optimize on a more complex and general function class of the objective has long been desired. Specifically, we consider modeling the input space for blackbox optimization in reproducing kernel Hilbert spaces (RKHS). This modeling leads to a functional optimization problem whose domain is a function space that enables us to optimize in a very rich function class. In addition, we propose CMAESRKHS, a generalized CMAES framework, that performs blackbox functional optimization in RKHS. A search distribution, represented as a Gaussian process, is adapted by updating both its mean function and covariance operator. Adaptive representation of the mean function and the covariance operator is achieved by resorting to sparsification. CMAESRKHS is evaluated on two simple functional optimization problems and two benchmark reinforcement learning (RL) domains. For an application in RL, we model policies for MDPs in RKHS and transform a cumulative return objective as a functional of RKHS policies, which can be optimized via CMAESRKHS. This formulation results in a blackbox functional policy search framework.
Related Material


