Shadow Dexterous Hand Control via MediaPipe and BioIK Integration

Abstract

This paper presents a study on the teleoperation of an anthropomorphic robotic hand. The proposed system integrates MediaPipe for human hand tracking and BioIK for translating the captured data to the robotic hand. Two methods for acquiring three-dimensional hand positions are evaluated: direct estimation via MediaPipe and a stereo vision-based technique. Experimental results reveal that the stereo technique significantly improves the accuracy and fidelity of the hand position replication, albeit at the cost of increased teleoperation instability. The algorithm achieves an end-to-end delay of one second, ensuring real-time usability within certain constraints. The system's effectiveness is further validated through successful object-grasping experiments, highlighting its potential for advanced teleoperation applications.