A Novel Model for Arbitration between Planning and Habitual Control\n Systems

Abstract

It is well established that humans decision making and instrumental control\nuses multiple systems, some which use habitual action selection and some which\nrequire deliberate planning. Deliberate planning systems use predictions of\naction-outcomes using an internal model of the agent's environment, while\nhabitual action selection systems learn to automate by repeating previously\nrewarded actions. Habitual control is computationally efficient but may be\ninflexible in changing environments. Conversely, deliberate planning may be\ncomputationally expensive, but flexible in dynamic environments. This paper\nproposes a general architecture comprising both control paradigms by\nintroducing an arbitrator that controls which subsystem is used at any time.\nThis system is implemented for a target-reaching task with a simulated\ntwo-joint robotic arm that comprises a supervised internal model and deep\nreinforcement learning. Through permutation of target-reaching conditions, we\ndemonstrate that the proposed is capable of rapidly learning kinematics of the\nsystem without a priori knowledge, and is robust to (A) changing environmental\nreward and kinematics, and (B) occluded vision. The arbitrator model is\ncompared to exclusive deliberate planning with the internal model and exclusive\nhabitual control instances of the model. The results show how such a model can\nharness the benefits of both systems, using fast decisions in reliable\ncircumstances while optimizing performance in changing environments. In\naddition, the proposed model learns very fast. Finally, the system which\nincludes internal models is able to reach the target under the visual\nocclusion, while the pure habitual system is unable to operate sufficiently\nunder such conditions.\n