Lyapunov-Informed Multi-Agent Reinforcement Learning for Multi-Robot Cooperation Tasks

摘要

Multi-Agent Reinforcement Learning (MARL) has shown great potential in solving complex tasks. Despite great success, low training efficiency remains a pervasive and long-standing challenge in MARL. To tackle this issue, it is promising to leverage prior knowledge or environmental properties to inform and improve the MARL. We notice that many multi-agent tasks specify certain goal states where special rewards are granted, guiding agents to achieve the goal. Inspired by the theory of Lyapunov stability, an intuitive optimal policy to the tasks should be able to asymptotically converge to the goal states from any initial, making the goal states stable equilibria. Focusing on this type of tasks, we introduce the concept of Lyapunov Markov game (LMG), a new subclass of the cooperative Markov game, featuring a set of goal states and goal-oriented reward function. We then provide a theoretical bound on scaled value distance as a necessary condition to obtain a stable suboptimal policy in LMG. Motivated by this insight, we further propose the Lyapunov-informed MARL, which leverages a newly-designed Lyapunov-informed reward. Theoretical work is conducted to show that the Lyapunov-informed MARL enjoys a broadened bound, facilitating the training process to find a stable suboptimal policy more easily and then converge to an optimal policy more efficiently. Extensive experiments and real-world multi-robot implementations are conducted to show the superior performance of the proposed approach over advanced baseline models.