Operation Interface of Surgical Assistant Robot Based on Image Recognition of Forceps Shaft

Abstract

With the collaboration of locally operated small surgical robot in a sterile area, a surgeon can perform robotically assisted laparoscopic solo surgery. A new operation interface based on image recognition of forceps performed by the surgeon to control a surgical assistant manipulator has been developed. The forceps shaft captured using RGB and depth images is extracted by Mask R-CNN, and the orientation of the shaft is estimated for the control command. The prediction on test images of actual operation for the Mask R-CNN trained model on simulated operation images was evaluated.