Improving Learning for Embodied Agents in Dynamic--Environments by State Factorisation

Abstract

A new reinforcement learning algorithm de-signed specifically for robots and embodied sys-tems is described. Conventional reinforcement learning methods intended for learning general tasks suffer from a number of disadvantages in this domain including slow learning speed, an in-ability to generalise between states, reduced per-formance in dynamic environments, and a lack of scalability. Factor-Q, the new algorithm, uses factorised state and action, coupled with mul-tiple structured rewards, to address these is-sues. Initial experimental results demonstrate that Factor-Q is able to learn as efficiently in dy-namic as in static environments, unlike conven-tional methods. Further, in the specimen task, obstacle avoidance is improved by over two or-ders of magnitude compared with standard Q-learning. 1.