Model-Based Offline Reinforcement Learning for Autonomous Delivery of Guidewire

Abstract

Guidewire delivery is a fundamental procedure in percutaneous coronary intervention. The inherent flexibility of the guidewire poses challenges in precise control, necessitating long-term training and substantial expertise. In response, this paper proposes a novel offline reinforcement learning (RL) algorithm, Conservative Offline Reinforcement Learning with Variational Environment Model (CORVE), for autonomous delivery of guidewire. CORVE first uses offline data to train an environment model and then optimizes the policy with both offline and model-generated data. The proposed method shares an encoder between the environmental model, policy, and Qfunction, mitigating the common sample inefficiency in imagebased RL. Besides, CORVE utilizes model prediction errors to forecast wrong deliveries in inference, which is an attribute absent in existing methods. The experimental results show that CORVE obtains superior performance in guidewire deliveries, achieving notably higher success rates and smoother movements than existing methods. These findings suggest that CORVE holds significant potential for enhancing the autonomy of vascular robotic systems in clinical settings.