Balancing Real-world Interaction and VR Immersion with AI Vision Robotic Arm

Abstract

With the widespread adoption of Virtual Reality (VR) technology, VR devices can be more realistic and make the tasks they perform feel real to the user. It is this sense of realism that makes more and more people are using VR headsets. While using VR devices, the need to interact with the real world will also exist. For example, when people are using a VR headset to study and work, they have a physiological need (thirst, hunger, etc.), and need to interact with reality. But the closed nature of the VR headset causes this interaction to be very disruptive to the immersion of the user's current task. To address this issue, we propose an interaction method that combines an AI vision algorithm and a robotic arm to preserve immersion during task performance in VR. This method addresses the challenge of interacting with the real world while using closed VR devices. Our experiments show that our method outperforms see-through and mapping methods, allowing users to quickly re-immerse themselves after interruptions and reducing interruptions during real-world task performance. Future research will focus on improving this method without sensitization and expanding the range of real-world operations that can be performed.