Digital twins for hand gesture-guided human-robot collaboration systems

Abstract

Gesture control is one of the effective and flexible communication method between humans and robots. However, it always depends on complex hardware and configurations in human-robot collaboration systems. Simplifying the design of gesture-interaction systems and avoiding miscommunication are challenging problems. In this paper, we proposed a method that utilizes an RGB sensor to realize spatial human-robot collaboration. A random forest based depth estimator is presented to supplement the additional spatial information for hand gesture recognition. Additionally, we demonstrate the construction of secure human-robot collaboration scenarios in Unity and validate our approach in real-world settings, based on which a digital twin system oriented to human-machine collaboration is constructed to realize rapid human-machine task simulation, safety specification testing, and real-scene applications development.