Overcoming Obstacles With a Reconfigurable Robot Using Deep Reinforcement Learning Based on a Mechanical Work-Energy Reward Function

Abstract

This paper presents a Deep Reinforcement Learning (DRL) method based on a mechanical (work) Energy reward function applied to a reconfigurable RSTAR robot to overcome obstacles. The RSTAR is a crawling robot that can reconfigure its shape and shift the location of its center of mass via a sprawl and a four-bar extension mechanism. The DRL was applied in a simulated environment with a physical engine (UNITY <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^{\mathrm {TM}}$ </tex-math></inline-formula> ). The robot was trained on a step obstacle and a two-stage narrow passage obstacle composed of a horizontal and a vertical channel. To evaluate the benefits of the proposed Energy reward function, it was compared to time-based and movement-based reward functions. The results showed that the Energy-based reward produced superior results in terms of obstacle height, energy requirements, and time to overcome the obstacle. The Energy-based reward method also converged faster to the solution compared to the other reward methods. The DRL’s results for all the methods (energy, time and movement- based rewards) were superior to the best results produced by the human experts (see attached video).