Reinforcement Learning in a Neurally Controlled Robot Using Dopamine\n Modulated STDP

摘要

Recent work has shown that dopamine-modulated STDP can solve many of the\nissues associated with reinforcement learning, such as the distal reward\nproblem. Spiking neural networks provide a useful technique in implementing\nreinforcement learning in an embodied context as they can deal with continuous\nparameter spaces and as such are better at generalizing the correct behaviour\nto perform in a given context.\n In this project we implement a version of DA-modulated STDP in an embodied\nrobot on a food foraging task. Through simulated dopaminergic neurons we show\nhow the robot is able to learn a sequence of behaviours in order to achieve a\nfood reward. In tests the robot was able to learn food-attraction behaviour,\nand subsequently unlearn this behaviour when the environment changed, in all 50\ntrials. Moreover we show that the robot is able to operate in an environment\nwhereby the optimal behaviour changes rapidly and so the agent must constantly\nrelearn. In a more complex environment, consisting of food-containers, the\nrobot was able to learn food-container attraction in 95% of trials, despite the\nlarge temporal distance between the correct behaviour and the reward. This is\nachieved by shifting the dopamine response from the primary stimulus (food) to\nthe secondary stimulus (food-container).\n Our work provides insights into the reasons behind some observed biological\nphenomena, such as the bursting behaviour observed in dopaminergic neurons. As\nwell as demonstrating how spiking neural network controlled robots are able to\nsolve a range of reinforcement learning tasks.\n