Modular Deep Reinforcement Learning for Emergent Locomotion on a Six-Legged Robot

Abstract

Deep Reinforcement Learning (DRL) approaches have shown tremendous success over the last years in different application areas. But control of robots in real world settings and when facing unpredictable environments has still proven to be a difficult task that requires unreasonable long training times. This has sparked new interest in the organization of animal and human motor control systems and how to transfer these insights into such DRL learning architectures. While a hierarchical organization has now been advocated and introduced into a couple of DRL approaches, we propose decentralization as one further effective organizational principle. As an example we are considering insect locomotion (in particular locomotion of stick insects) which is adaptive and robust even when dealing with unpredictability. The underlying control system is assumed to consist of six individual local control modules that each control the action of a single leg. These local controllers only share limited information with neighboring legs which has shown sufficient to produce robust adaptive walking. In this article, we implement such a decentralized architecture of six local control modules and train it using Deep Reinforcement Learning. This architecture shows faster training times compared to a standard centralized approach and a trend towards more adaptive behavior and better performance when facing uncertain environments.