MuViH: Multi-View Hand gesture dataset and recognition pipeline for human–robot interaction in a collaborative robotic finishing platform

Abstract

The proliferation of tedious and repetitive tasks on production lines has accelerated the deployment of automated robots. This has also led to a demand for more flexible robots, known as cobots, that can work in collaboration with operators to perform a variety of tasks in different contexts. This paper explores the potential of computer vision-based hand gesture recognition as a means of human–robot interaction within cobotic platforms. Our research focuses on the challenges of gesture recognition in the face of visual occlusions and different camera viewpoints, typical of part finishing tasks in a real-world industrial setting. We introduce a new dataset, MuViH (Multi-View Hand gesture), which features a high variability in camera viewpoints, human operator characteristics, and occlusions, and is fully annotated for hand detection and gesture recognition. We then present a comprehensive hand gesture recognition pipeline that leverages this dataset. Our pipeline incorporates a multi-view aggregation step that significantly enhances gesture recognition accuracy, particularly in the case of visual occlusions. Thanks to extensive experiments and cross-validation on the MuViH dataset and another public dataset, HANDS, our approach demonstrates state-of-the-art performance in gesture recognition. This breakthrough underlines the potential of integrating robust vision-based interaction techniques into cobotic systems, improving flexibility and speed on the production line. • MuViH dataset includes over 85,000 images for multi-view hand gesture recognition. • MuViH offers high variability in camera viewpoints, human features and occlusions. • MuViH is fully annotated for hand detection and static gesture recognition. • The proposed pipeline shows SOTA performances for hand detection and gesture recognition. • A multi-view version of the pipeline improves by 14% the gesture recognition accuracy.