Curriculum-based reinforcement learning for path tracking in an underactuated nonholonomic system

Abstract

Underactuated mechanical systems with nonholonomic constraints find applications in bioinspired robotics, such as snake-like robots and more recently in fish-like aquatic robots. Animal locomotion suggests that in such bioinspired robots, gaits or cyclic changes in kinematics or shape variables lead to efficient and agile motion. Path tracking in such nonholonomic systems that are not purely kinematic can be a challenging problem. In this paper we consider the problem of path tracking by a modified Chaplygin sleigh with a ‘tail’ which is a four degree of freedom nonholonomic system, possessing a single internal reaction wheel as an actuator. We develop a curriculum based deep Reinforcement Learning (RL) optimal control approach for simultaneous velocity and path tracking for this system. The curriculum based learning approach first leads to a policy for optimal tracking of limit cycles in a reduced velocity space and then in a next step to track a path. This curriculum approach allows an RL agent to learn the ’mechanics on invariant manifolds’ of the system and can be a useful approach in the motion control of high degree of freedom robots with increasing model complexity.