Hand Action Perception and Robot Instruction

Abstract

It is extremely popular and much more efficient way for teachers or instructors at school, work, or play, to teach their students a task just by demonstrating the task clearly in front of the students. Then the students replicate the demonstrated task to verify if they are able to do it successfully. Based on this observation, we presents a general and robust approach to hand action perception for automatic robot instruction using depth image sequences. We aim to build such a robot system which should learn not only the initial poses and final destinations of the objects and the orders of assembly, but also should perceive how objects move and how they are manipulated. In our system the human instructor must simply demonstrate an assembly task in front of a vision system in the human world, no dataglove or special markings are necessary. The recorded image sequences are used to recover a depth image sequence for model-based human hand and object tracking to form the perceptual data st...