dVPose: Automated Data Collection and Dataset for 6D Pose Estimation of Robotic Surgical Instruments

Abstract

We present dVPose, a realistic multi-modality dataset intended for use in the development and evaluation of real-time single-shot deep-learning based 6D pose estimation algorithms on a head mounted display (HMD). In addition to the dataset, our contribution includes an automated (robotic) data collection platform that integrates an accurate optical tracking system to provide the ground-truth poses. We collected a comprehensive set of data for vision-based 6D pose estimation, including images and poses of the extra-corporeal portions of the instruments and endoscope of a da Vinci surgical robot. The images are collected using the multi-camera rig of the Microsoft HoloLens 2 HMD, mounted on a UR10 robot, and the corresponding poses are collected by optically tracking both the instruments/endoscope and HMD. The intended application is to enable markerless localization of the HMD with respect to the da Vinci robot, considering that the instruments and endoscope are among the few robotic components that are not covered by sterile drapes. Our dataset features synchronized images from the RGB, depth, and grayscale cameras of the HoloLens 2 device. It is unique in that it provides medically focused images, provides images from a HoloLens 2 device where object tracking is a fundamental task, and provides data from multiple visible-light cameras in addition to depth. Furthermore, the automated data collection platform can be easily adapted to collect images and ground-truth poses of other objects.