Using Parameterized Black-Box Priors to Scale Up Model-Based Policy\n Search for Robotics

Abstract

The most data-efficient algorithms for reinforcement learning in robotics are\nmodel-based policy search algorithms, which alternate between learning a\ndynamical model of the robot and optimizing a policy to maximize the expected\nreturn given the model and its uncertainties. Among the few proposed\napproaches, the recently introduced Black-DROPS algorithm exploits a black-box\noptimization algorithm to achieve both high data-efficiency and good\ncomputation times when several cores are used; nevertheless, like all\nmodel-based policy search approaches, Black-DROPS does not scale to high\ndimensional state/action spaces. In this paper, we introduce a new model\nlearning procedure in Black-DROPS that leverages parameterized black-box priors\nto (1) scale up to high-dimensional systems, and (2) be robust to large\ninaccuracies of the prior information. We demonstrate the effectiveness of our\napproach with the "pendubot" swing-up task in simulation and with a physical\nhexapod robot (48D state space, 18D action space) that has to walk forward as\nfast as possible. The results show that our new algorithm is more\ndata-efficient than previous model-based policy search algorithms (with and\nwithout priors) and that it can allow a physical 6-legged robot to learn new\ngaits in only 16 to 30 seconds of interaction time.\n