A framework for reinforcement learning on real robots

Abstract

Robots are now being used in complex, unstructured environments, performing ever more sophisticated tasks. As the task and environmental complexity increases, the need for effective learning on such systems is becoming more and more apparent. Robot programmers often find it difficult to articulate their knowledge of how to perform a given task in a form suitable for robots to use. Even when they can, the limitations of robot sensors and actuators might render their intuitions less effective. Also, it is often not possible to anticipate (and code for) all environments in which the robot might find itself having to perform a certain task. Therefore, it seems useful to have the robot be able to learn to act, in the hope of overcoming these two difficulties. One promising approach to learning on real robots that is attracting considerable interest at the moment is reinforcement learning. The programmer must supply a reward function which maps states of the world onto a scalar reward, essentially saying how good or bad it is to be in a given state. Once given this function, the robot can, in theory, learn an appropriate action policy to maximize some measure of reward over time. Designing such a function seems more intuitive than writing code to perform the task, since we can do so in more general terms. However, there are some significant problems when we attempt to use this strategy on a real, physical robot, including (but not limited to) the following.