Sketch RL: Interactive Sketch Generation for Long-Horizon Tasks via Vision-Based Skill Predictor

Abstract

For autonomous robots, it is desirable to learn coordination of primitive skills that can effectively solve long-horizon tasks and perform novel ones. Recent advances in hierarchical policy learning have shown that decomposing complex tasks into sequences of primitive skills which are called sketches can enable robots to perform directed exploration in challenging manipulation tasks. However, they usually fall short in sequencing skills in a new task without retraining as the task sketches are almost hard-coded or learned by deep reinforcement learning. To improve exploration efficiency for long-horizon tasks, we propose Sketch RL, a hierarchical framework that combines supervised learning with reinforcement learning interactively generates the task sketch, and utilizes it as the curriculum to guide low-level skill learning. Furthermore, to allow for multitask decomposition and generalizing few-shot to new tasks, our method exploits a Vision-based Skill Predictor (VSP) to capture shared subtask structure. Extensive experiments on challenging manipulation tasks demonstrate that Sketch RL substantially outperforms other prior baseline methods and is capable of adapting to new tasks with different sketches and real-world settings.