Trajectory-Tracking Control of Robotic System via Proximal Policy Optimization

Abstract

This paper aims to solve the trajectory tracking problem of manipulator and mobile robot by using proximal policy optimization (PPO) with generalized advantage estimation (GAE). We adopt a distributed framework of PPO to improve the speed of sample collection and reduce the correlation of transitions when updating the networks. Random reference state initialization is used during the training process to enable the robot to learn effectively from the reference trajectory. Additionaly, we introduce Long Short-term Memory (LSTM) to represent the actor and critic, and buffers to store the cell state and hidden state of LSTM for the initialization of each episode, which helps solve the problem of inaccurate inital LSTM states. Distributed PPO with fully-connected neural networks and LSTM is utilized to train the manipulator and mobile robot with an exponential reward function to track the given trajectories. Numerical simulations are provided to demonstrate the effectiveness of the proposed method. It can be seen that distributed PPO with LSTM can improve the tracking performance.