Hand Gesture Recognition To Understand Musical Conducting Action

Abstract

This paper deals with the understanding of four musical time patterns and three tempos that are generated by a human conductor of robot orchestra or an operator of computer-based music play system using the hand gesture recognition. We use only a stereo vision camera with no extra special devices such as sensor glove, 3D motion capture system, infra-red camera, electronic baton and so on. We propose a simple and reliable vision-based hand gesture recognition using the conducting feature point (CFP), the motion-direction code, and the motion history matching. The proposed hand gesture recognition system operates as follows: First, it extracts the human hand region by segmenting the depth information generated by stereo matching of image sequences. Next, it follows the motion of the center of the gravity(COG) of the extracted hand region and generates the gesture features such as CFP and the direction-code. Finally, we obtain the current timing pattern of the music's beat and tempo by the proposed hand gesture recognition using either CFP tracking or motion histogram matching. The experimental results show that the musical time pattern and tempo recognition rate are over 86% on the test data set when the motion histogram matching is used.