Hand Gesture Communication using Deep Learning based on Relevance Theory

Abstract

In recent years, social robots are widely research due to the advancement of electronics hardware and deep learning algorithms breakthrough. Hand gesture recognition playing an important role in human-robot interaction. However, to deploy state-of-art of deep learning methods for hand gesture recognition, researchers have to associate all hand gestures with their respective meanings (label). Re-train the deep learning model is required if a new gesture added for recognition. In addition, the meaning of hand gestures often depending on the environmental conditions. In this paper, we propose a framework that learns and recognizes hand gesture meaning based on surrounding objects. The proposed method consists of two sub frameworks: i) A deep learning model, Mask RCNN to detect and extract features of objects and hand gestures; ii) An incremental learning model, Declarative Memory Recurrent Neural Model to incrementally learn hand gestures meaning based on surrounding objects. The effectiveness of our proposed method is validated through several experiments.