Monocular Tracking of Human Hand on a Smart Phone Camera using MediaPipe and its Application in Robotics

Abstract

With Industry 4.0, robots are finding their way into the production lines of multiple industries. The conventional methods to program these robots require special training, help from a robot technician, or have a basic understanding of robot kinematics. Integration of machine learning into robotics can make the programming process shorter, cost-effective and user friendly. In this paper, we propose to use a customizable machine learning solution of ‘Google MediaPipe Hands' to track human hands from a monocular camera of a smartphone and use it for robotic applications. MediaPipe Hands provides only a 2.5D pose estimation; we propose to use a simple calibration and the concept of perspective projection to get the 3D position of the hands relative to the smartphone. By conducting multiple experiments, we found that the hand tracking solution has satisfactory accuracy rates. We developed robot simulations in Python to check the viability of the hand tracker for robotic applications. We could accurately control the end-effector movement using the hand tracker.