Self-Supervised Surgical Tool Segmentation using Kinematic Information

Abstract

Surgical tool segmentation in endoscopic images is the first step towards\npose estimation and (sub-)task automation in challenging minimally invasive\nsurgical operations. While many approaches in the literature have shown great\nresults using modern machine learning methods such as convolutional neural\nnetworks, the main bottleneck lies in the acquisition of a large number of\nmanually-annotated images for efficient learning. This is especially true in\nsurgical context, where patient-to-patient differences impede the overall\ngeneralizability. In order to cope with this lack of annotated data, we propose\na self-supervised approach in a robot-assisted context. To our knowledge, the\nproposed approach is the first to make use of the kinematic model of the robot\nin order to generate training labels. The core contribution of the paper is to\npropose an optimization method to obtain good labels for training despite an\nunknown hand-eye calibration and an imprecise kinematic model. The labels can\nsubsequently be used for fine-tuning a fully-convolutional neural network for\npixel-wise classification. As a result, the tool can be segmented in the\nendoscopic images without needing a single manually-annotated image.\nExperimental results on phantom and in vivo datasets obtained using a flexible\nrobotized endoscopy system are very promising.\n