Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning\n and Large-Scale Data Collection

Abstract

We describe a learning-based approach to hand-eye coordination for robotic\ngrasping from monocular images. To learn hand-eye coordination for grasping, we\ntrained a large convolutional neural network to predict the probability that\ntask-space motion of the gripper will result in successful grasps, using only\nmonocular camera images and independently of camera calibration or the current\nrobot pose. This requires the network to observe the spatial relationship\nbetween the gripper and objects in the scene, thus learning hand-eye\ncoordination. We then use this network to servo the gripper in real time to\nachieve successful grasps. To train our network, we collected over 800,000\ngrasp attempts over the course of two months, using between 6 and 14 robotic\nmanipulators at any given time, with differences in camera placement and\nhardware. Our experimental evaluation demonstrates that our method achieves\neffective real-time control, can successfully grasp novel objects, and corrects\nmistakes by continuous servoing.\n