Augmented Reality Enabled Telepresence Framework for Intuitive Robotic Teleoperation

Abstract

In terms of telepresence for robotic teleoperation, the existing remote surgery provides the patient’s side real-time videos to remote operators. Its effectiveness is limited due to the lack of depth information, occlusion, and restricted viewing angles. To address it, this paper develops a robotic telepresence framework integrating augmented reality (AR) and 3D reconstruction technology, providing a 3D virtual patient’s side environment for remote operators. The real scene of the patient’s side is reconstructed by a monocular camera with a manipulator before the operation, and the manipulator is teleoperated by the operator at the remote’s side. The manipulator’s pose is fused into the classic SVO+REMODE algorithm to improve the reconstructed accuracy. On the remote’s side, an AR head-mounted display (HMD) is deployed to provide telepresence for operators under a similar environment, and the reconstructed scene of the patient’s side is registered to the remote room, from which the operator can see the whole scene of the patient body and surrounding objects. An ArUco cube is used to evaluate the superimposed error between virtual and real displays, and the effectiveness is validated in an experimental platform for remote surgery.