Transformer-based Multi-Agent Reinforcement Learning for Generalization of Heterogeneous Multi-Robot Cooperation

Abstract

Recent advances in multi-agent reinforcement learning (MARL) have significantly enhanced cooperation capabilities within multi-robot teams. However, the application to heterogeneous teams poses the critical challenge of combinatorial generalization—adapting learned policies to teams with new compositions of varying sizes and robots capabilities. This challenge is paramount for dynamic real-world scenarios where teams must swiftly adapt to changing environmental and task conditions. To address this, we introduce a novel transformer-based MARL method for heterogeneous multirobot cooperation. Our approach leverages graph neural networks and self-attention mechanisms to effectively capture the intricate dynamics among heterogeneous robots, facilitating policy adaptation to team size variations. Moreover, by treating robot team decisions as sequential inputs, a capability-oriented decoder is introduced to generate actions in an auto-regressive manner, enabling decentralized decision-making that tailored each robot’s varying capabilities and heterogeneity type. Furthermore, we evaluate our method across two heterogeneous cooperation scenarios in both simulated and real-world environments, featuring variations in team number and robot capabilities. Comparative results reveal our method’s superior generalization performance compared to existing MARL methodologies, marking its potential for real-world multi-robot applications.