Learn From Safe Experience: Safe Reinforcement Learning for Task Automation of Surgical Robot

摘要

Surgical task automation in robotics can improve the outcomes, reduce quality-of-care variance among surgeons and relieve surgeons’ fatigue. Reinforcement learning (RL) methods have shown considerable performance in robot autonomous control in complex environments. However, existing RL algorithms for surgical robots do not consider any safety requirements, which is unacceptable in automating surgical tasks. In this work, we propose an approach called Safe Experience Reshaping (SER) that can be integrated into any offline RL algorithm. Firstly, the method identifies and learns the geometry of constraints. Secondly, a safe experience is obtained by projecting an unsafe action to the tangent space of the learned geometry, which means that the action is in the safe space. Then, the collected safe experiences are used for safe policy training. We designed 3 tasks that closely resemble real surgical tasks including 2D cutting tasks and a contact-rich debridement task in 3D space to evaluate the safe RL framework. We compare our framework to 5 state of the art RL methods including reward penalty and primal-dual methods. Results show that our framework gets a lower rate of constraint violations and better performance in task success, especially with a higher convergence speed.