A new two-layer reinforcement learning approach the control of a 2DOF manipulator

Abstract

This paper presents a new machine learning approach, based on reinforcement learning, to control a highly nonlinear robot demonstrator. Learning is achieved using an on-policy temporal difference learning agent framework. The agent controls the movements of the robot by choosing torques for each joint and immediately receives a feedback signal. The implemented SARSA-agent uses an update rule that calculates the estimate of the current state-action by using the current state-action value, the received reward and the following state-action value. To handle the high number of state-action value pairs, a hash-table is used to efficiently store and access these values inside the lookup-table. To accelerate the learning process of more complex motions, a new second layer approach is introduced. In this approach, a library of simple motions is created in the first layer. The second layer-agent then combines the gathered experiences to achieve a faster solution for more complex motions. The evaluation of the two-layer agent shows that the combination of both layers dramatically increases the speed of finding a solution. Additionally, its solution is often better than the solution found by the pure reinforcement learning agent.