Integration of Indonesian Speech and Hand Gesture Recognition for Controlling Humanoid Robot

Abstract

This paper proposes the integration method of Indonesian speech and hand gesture recognition for controlling humanoid robot. A Microsoft Kinect v2 is used to recognize human speech and gesture so that the humanoid robot can follow the command. The recognition of speech grammar is done by creating Indonesian language model which is converted from a combination of English lexicon-grammar. The method of gesture recognition uses Support Vector Machine (SVM) with Directed Acyclic Graph (DAG) decision. Then, the result is obtained from a combined score for all speech and gesture recognition using dynamic programming. Compare to the speech or gesture recognition only, the experimental results prove that the integration of Indonesian speech and gesture recognition has a better accuracy. The speech and gesture commands being tested are forward, backward, turn left, turn right, stop, sit down, and stand up. The testing was conducted by six different people with different gender and age where the distance between the Kinect and the human is around 1,5 meters. The rates of average accuracy of speech recognition, gesture recognition, and integration of speech and gesture are 80.71%, 92.74%, 94.05%, respectively.