Intelligent Human-Machine Interaction Based on Digital Twin and Virtual Reality

Abstract

The advent of Industry 4.0 has significantly promoted the field of intelligent manufacturing. New technologies are emerging to enable intelligent manufacturing. However, concerns arose on how to better apply these new technologies to manufacturing systems, and how to achieve better cooperation among them. In this paper, an intelligent human-machine interaction was proposed by taking a multi-axis robot arm as the research object; this is system, intelligent elements, such as virtual reality (VR) technology, artificial intelligence (AI) algorithm and digital twin, were collaborated. Specifically, a digital twin system was constructed to connect the digital and physical worlds, and virtual reality technology was used to achieve human-machine interaction between humans and digital models. We constructed a digital twin system using sensor data acquisition and digital modeling technologies, which achieved real-time motion synchronization between the physical and virtual bodies. To enhance the human-machine interaction's performance, this paper also introduced image recognition algorithms and reinforcement learning algorithms to its target recognition and grasping tasks, respectively. We use image recognition algorithm to identify the target and propose MAPPO- IK reinforcement learning algorithm to train and control the manipulator's motion. Integrating artificial intelligence algorithms and virtual reality into the digital twin system, this paper proposed an intelligent human-machine interaction approach. Our findings suggest that an integrated system combining virtual reality, artificial intelligence, and digital twin can significantly enhance human-machine interaction performance. Operators using this intelligent human-machine approach achieved faster learning speeds and higher fault tolerance. These results provide valuable insights for researchers exploring the potential of digital twin and virtual reality for intelligent human-machine interaction.