Mixed Reality and Digital Twin Interface for Robot Collaborative Tasks

Abstract

Traditional bimanual user interfaces and real-time video feeds are commonly used for robot control but become increasingly complex when managing multiple users and robots in collaborative tasks. Designing intuitive interfaces for multi-arm robotic systems is particularly challenging, especially in scenarios where direct in-situ control is impractical due to biohazards or stringent sterilization requirements. A Digital Twin (DT) offers a promising solution for monitoring and enabling interactive control of robotic systems in such environments. This work introduces a Mixed Reality (MR) interface that leverages the DT to provide operators with an intuitive and immersive means of controlling multi-arm robotic systems, enhancing efficiency and collaboration in complex robotic tasks.