Development of a Deep Deterministic Policy Gradient (DDPG) Algorithm for Suturing Task Automation

Abstract

Robot-Assisted Minimally Invasive Surgery (RAMIS) has brought in the last decades major improvements in terms of quality of operations, execution times, patient safety, and postop rehabilitation. Suturing, however, remains one of the most complex and time-consuming tasks. In this work, we propose a method that uses a Reinforcement Learning (RL) algorithm, the Deep Deterministic Policy Gradient (DDPG) [3], to generate optimal trajectories for the needle to follow, with the aim to automate, through the use of the da Vinci® surgical system, the execution of needle insertion into the tissue. This method, along with the standard Replay Buffer, involves the use of the Demonstration Buffer, consisting of optimal sequences of states and actions for the agent to achieve, in order to successfully complete the task. In addition, we trained the agent to accomplish more than one goal, through a multiple-goal sparse reward.