Motion tracking of both hands with occasional mutual occlusion using RGB-D camera and IMU

Abstract

Motion tracking of both hands is of paramount for human-robot interaction, in fields of telerobotics, learning from demonstration and motion sensing games, and so forth. Full coverage and high precision are challenging features for a motion tracking system, owing to occasional mutual occlusion and accumulated errors. In this paper, we present a low-cost hand motion tracking system integrating RGB-D camera and IMU and its corresponding algorithms. A new descriptor constructed with synthetic acceleration of both hands, from the RGB-D camera and from the IMU respectively, is proposed. This descriptor is then used to classify the current hand state into occlusive or non-occlusive by a support vector machine. Tracking results are finally obtained by fusing data from the RGB-D camera and the IMU with Kalman Filter and varying the covariance of measurement noise if occlusion detected. Experiments are conducted to verify the effectiveness of the proposed tracking system and its corresponding algorithms.