DDCO: Discovery of Deep Continuous Options for Robot Learning from\n Demonstrations

摘要

An option is a short-term skill consisting of a control policy for a\nspecified region of the state space, and a termination condition recognizing\nleaving that region. In prior work, we proposed an algorithm called Deep\nDiscovery of Options (DDO) to discover options to accelerate reinforcement\nlearning in Atari games. This paper studies an extension to robot imitation\nlearning, called Discovery of Deep Continuous Options (DDCO), where low-level\ncontinuous control skills parametrized by deep neural networks are learned from\ndemonstrations. We extend DDO with: (1) a hybrid categorical-continuous\ndistribution model to parametrize high-level policies that can invoke discrete\noptions as well continuous control actions, and (2) a cross-validation method\nthat relaxes DDO's requirement that users specify the number of options to be\ndiscovered. We evaluate DDCO in simulation of a 3-link robot in the vertical\nplane pushing a block with friction and gravity, and in two physical\nexperiments on the da Vinci surgical robot, needle insertion where a needle is\ngrasped and inserted into a silicone tissue phantom, and needle bin picking\nwhere needles and pins are grasped from a pile and categorized into bins. In\nthe 3-link arm simulation, results suggest that DDCO can take 3x fewer\ndemonstrations to achieve the same reward compared to a baseline imitation\nlearning approach. In the needle insertion task, DDCO was successful 8/10 times\ncompared to the next most accurate imitation learning baseline 6/10. In the\nsurgical bin picking task, the learned policy successfully grasps a single\nobject in 66 out of 99 attempted grasps, and in all but one case successfully\nrecovered from failed grasps by retrying a second time.\n