Hands and Faces, Fast: Mono-Camera User Detection Robust Enough to Directly Control a UAV in Flight

Abstract

We present a robust real-time system for simultaneous detection of hands and faces in RGB and gray-scale images, and a novel dataset used for training. Our goal is to provide a robust sensor front-end suitable for real-time human-robot interaction using face-engagement and gestures. Using hand-labelled videos obtained from real human-UAV interaction experiments, we re-trained the YOLOv2 Deep Convolutional Neural Network to detect only hands and faces. This model was then used to automatically label several much larger third-party datasets. After manual correction of these results, we modified and re-trained the model on all this labelled data. We obtain qualitatively good detection results at 60Hz on a commodity GPU: our simultaneous hand-and-face detector gives state of the art accuracy and speed in a hand detection benchmark and competitive results in a face detection benchmark. To demonstrate its effectiveness for human-robot interaction we describe its use as the input to a simple but practical gestural human-UAV interface for entertainment or industrial applications. All software, training and test data are freely available.