A Data-Efficient Reinforcement Learning Method Based on Local Koopman Operators

Abstract

Although providing exceptional asymptotic performance, trial-and-error based reinforcement learning suffers from being data inefficient, i.e., requiring an excessive amount of trial data for training. In this paper, we propose a data-efficient model-based reinforcement learning algorithm based on the Koopman operator theory. By representing the environment dynamics as a linear dynamical system in a high-dimensional space, the Koopman operator theory allows incorporating effective optimal control methods to produce high-quality trials thus accelerating learning. However, when applying the theory for reinforcement learning, with the sparse and unevenly distributed trial data, it is difficult to learn globally linear representations thus leading to serious model bias. To overcome this problem, we devise a local Koopman operator approach that is tailored for the setup of reinforcement learning. By coupling with deep neural networks and the linear quadratic regulator control, we propose the first Koopman operator model-based reinforcement learning algorithm called deep Koopman reinforcement learning (DKRL). The simulation experiment results show the proposed method can outperform existing approaches and can learn robotics control tasks with a small amount of data.