Fast Reinforcement Learning for Three-Dimensional Kinetic Human–Robot Cooperation with an EMG-to-Activation Model

Abstract

Kinetic human–machine cooperation has been investigated in research fields such as power assist and rehabilitation. Electromyographic (EMG) signals have often been used for this purpose instead of force sensors, since the EMG signals reflect the motor intention of a user, observed prior to actual movements, leading to more natural interaction. However, an inherent problem in using EMG signals is their time-varying nature caused by the fact that muscle coordination can vary over time because of differences between the closed-loop feedback systems in the calibration stage and in the actual task. This paper proposes the use of a policy gradient type of reinforcement learning for overcoming this problem by formulating EMG-based kinetic human–robot cooperative tasks as goal-oriented tasks, in which the force exerted by the user for kinetic interaction is estimated to achieve a goal shared with the robot. The reinforcement learning enables the force estimator to be adaptive to the time-varying nature of the EMG signals. The force estimator is based on the so-called EMG-to-activation model which is biologically plausible and has only a small number of parameters, enabling fast learning. A three-dimensional cooperative transfer task demonstrates the feasibility of our approach.