Reinforcement Learning for Control of Human Locomotion in Simulation

Abstract

Control of robotic leg prostheses and exoskeletons is an open challenge. Computer modeling and simulation can be used to study the dynamics and control of human walking and extract principles that can be programmed into robotic legs to behave similar to biological legs. In this study, we present the development of an efficient two-layer Q-learning algorithm, with k-d trees, that operates over continuous action spaces and a reward model that estimates the degree of muscle activation similarity between the agent and human state-to-action pairs and state-to-action sequences. We used a human musculoskeletal model acting in a high-dimensional physics-based simulation environment to train and evaluate our algorithm to simulate biomimetic walking. We used imitation learning and artificial biomechanics data to accelerate training via “expert” demonstrations and used experimental human data to compare and validate our predictive simulations, achieving 79% accuracy. Moreover, when compared to the previous state-of-the-art that used deep deterministic policy gradient, our algorithm was significantly more efficient with lower computational and memory storage requirements (i.e., requiring 7 times less RAM and 87 times less CPU compute), which can benefit real-time embedded computing for robot control. Overall, our new two-layer Qlearning algorithm using sequential data for continuous imitation of human locomotion serves as a first step towards the development of bioinspired controllers for robotic prosthetic legs and exoskeletons. Future work will focus on improving the prediction accuracy compared to experimental data and expanding our simulations to other locomotor activities.