Robust hand tracking with refined CAMShift based on combination of Depth and image features

Abstract

Hand tracking is essential for natural Human Robot/Computer Interaction (HRI/HCI), although efficient and robust hand tracking in complex environment is still a challenging issue. While most researchers simplify the issue by strictly controlling the environment with many restrictions on users' clothing, or the scene complexity, or hand motion, this paper focused on reducing these restrictions. As one major cause of the restrictions is the lack of depth information, this paper proposed a method combining depth cues with image features. Depth and motion cues were extracted through background subtraction and histogram based segmentation. Guided by the depth cues extracted, color image features were then extracted with skin-color region segmentation. Then different cues were fused adaptively to construct a probability map for the hand to be tracked. With this map, a refined CAMShift tracking scheme was developed. And based on hand direction constraints we conjectured empirically, a further refinement step was proposed to segment hand from forearm, which is usually avoided using restrictions on clothing for simplicity. A number of experiments were performed to demonstrate the method's effectiveness and robustness. Tracking rates in the experiments are around 85% for ordinary situations, and around 75% for complex situations, such as fast hand motion and distractors.