Smart Glove and Hand Gesture-based Control Interface For Multi-rotor Aerial Vehicles

Abstract

This paper introduces an adaptable human-robot interface that uses two types of human-computer interactions: an image processing technique for a right-hand gesture recognition and a smart glove for left-hand commands. A fixed number of gestures is used for specific commands to the vehicle (takeoff, land, hover, etc.), while the smart glove is used for the vehicle motors control. A single shot multi-box detector (SSD) model is used for a hand detection. After removing the cluttered background, the region of interest (RoI) is fed to a convolutional neural network (CNN) for right-hand gesture recognition. We propose three concurrent validation layers including a human-based validation. The validation layers allow the system to adapt to various users including different skin colors and hand shapes. Four flex sensors and a motion processing unit (MPU) are used in the smart glove to measure the bending ratio of each finger and the roll angle of the left hand. These signals are used for a left-hand gesture recognition as well as generation of continuous control signals such as throttle and angle commands of the vehicle. Extensive experimental results are presented that validate the proposed control methods.