Learning by Experience and by Imitation in Multi-Robot Systems

Abstract

A static strategy is defined during the design of the robotic system and after that, it is applied to the robots by choosing roles and actions of each robot depending of the a priori defined situation. A disadvantage of this kind of strategy is that it doesn't adapt automatically to changes in requirements and can lead to a low performance of the system if situations not envisaged by the designers occur. On the other hand, a dynamic strategy adapts itself to the environment. This kind of strategy is generally implemented with artificial intelligence techniques. Dynamic strategies can be divided in two stages: the learning and the using stage. In the learning stage, the overall system is exposed to simulated environments, where, if necessary, opponents are programmed using static strategies. In the using stage, the system does not modify the strategy parameters. Both stages can be executed one after another during all useful life of the system. This traditional way of implementing dynamic strategies requires a predefined set of possible situations and actions. Thus, because robots already know the kind of situations they can find in the environment and also they know what actions they can execute, we denominate these traditional approaches as "learning by experience". We conjecture that, in real world and not in minimal applications, the robots can not completely know what kind of situations they can find and also what are all the actions that they can perform, in this case, consider an action as a set of consecutive low level signals to the actuators of the robots. In this sense, we propose to combine the imitation learning approach with learning by experience in order to construct robots that really adapt to environment changes and also evolve during their useful life. Imitation learning can help the robots to know new actions and situations where it can be applied and learning by experience can help robots to test the new actions in order to establish if they really work for the whole team. It is important to note that the concepts, algorithms, and techniques proposed and evaluated in this work are focused in the task execution stage, specifically in dynamical strategies. Also, all the concepts are valid for multi-robot and multi-agent systems. The algorithms traditionally used for implementing learning by experience approaches are explained in section 2. Between them, we choose reinforcement learning algorithms for testing our model. In section 3 we explain our approach for implementing imitation learning and in section 4 we explain how the overall process of learning is implemented. Because there are several ways or paradigms for applying reinforcement learning to multi-robot systems , we compare them in section 5. Also, results obtained when the overall process was applied to a robot soccer problem are discussed in section 6. Finally conclusions of this work are explained in section 7.