Visual hand posture tracking in a gripper guiding application

Abstract

This paper proposes a real-time visual hand tracking and posture estimation system to guide a robotic arm in gripping gestures. We present an original 26 degree-of-freedom model of the hand, for which forward and inverse kinematics formulations have been developed. The hand, wearing a dark glove marked with colored cues, is tracked in real time in the image while completing a grasping task. The position of the color markers in the image is used to reconstruct the hand's pose. Occlusions are handled by position prediction and validation. A robot gripper can then be guided by the different configurations of the hand with respect to the object to be grasped, provided the relationship between the hand and the gripper is known. Successful results for the teleoperated control of a robot manipulator in simulation are presented.