Fast Reactive Grasping with In-Finger Vision and In-Hand FPGA-accelerated CNNs

摘要

We present a soft humanoid hand with in-finger integrated cameras and in-hand real-time image processing system for fast reactive grasping. Specifically, we describe an FPGA-based, in-hand integrated, embedded system for processing visual data captured by the five in-finger cameras while avoiding high bandwidth raw data streaming via the robots real-time data bus. The hardware acceleration allows fast detection and localization of objects based on finger-camera images and provides input for a grasping controller. To this end, we implement a resource-aware encoder-decoder Convolutional Neural Network (CNN) for pixel-wise object segmentation and run inference on the in-hand embedded system at 3.58 GOPS. We evaluate the system, consisting of the soft hand with in-finger vision and the in-hand FPGA-accelerated CNN in several experiments on the humanoid robot ARMAR-6. Specifically, we evaluate the overall system response time, the ability to perform precision grasps and test reactivity and reliability that are required for handover actions. We obtain an overall system response time of 154 ms for catching a falling object and obtain a success rate of 90 % reliability for the power drill handover tasks. Further, we successfully demonstrate ability of dexterous grasping and manipulation of a pencil from a cup.