Robotic surgery remote mentoring via AR with 3D scene streaming and hand interaction

Abstract

With growing popularity of robotic surgery, education becomes increasingly important and urgently needed. However, experienced surgeons have limited accessibility due to busy clinical schedules or working in a distant city, thus can hardly provide sufficient education resources for novices. Remote mentoring, as an effective way, can solve this problem, but traditional methods are limited to plain text, audio, or 2D video, which are not intuitive nor vivid. Augmented reality (AR) offers new possibilities for interactive teaching. In this paper, we propose a novel AR-based robotic surgery remote mentoring system with efficient 3D scene visualisation and natural hand interaction. Using a head-mounted display, mentors can remotely monitor the procedure streamed from trainees’ operation side. Mentors can also provide feedback directly with hand gestures, which is transmitted to trainees and viewed in robot console as guidance. We comprehensively validate the system on both real surgery videos and ex-vivo training tasks (peg-transfer and suturing). Promising results are demonstrated regarding fidelity of streamed scene visualisation, accuracy of feedback with hand interaction, and low-latency of each component in the system. This work showcases the feasibility of leveraging AR for reliable, flexible and low-cost solutions to robotic surgical education, and holds great potential for clinical applications.