Evaluations of embedded modules dedicated to multimodal human-robot interaction

Abstract

With personal robotics and assistance to dependant people, robots are in continuous interface with humans. To enable a more natural communication, based on speech and gesture, robots must be endowed with auditive and visual perception capacities. This paper describes a modular multimodal interface based on speech and gestures in order to control an interactive robot called Jido. In this paper we describe how the speech recognition and understanding module processes deictic and anaphoric utterances in a robust way and how gesture is taken into account at the fusion level to complement speech utterances. These two other modules, namely gesture interpretation and probabilistic multimodal fusion are then briefly depicted. Their integration and the associated robotics experiments are finally reported. These experiments were carried out in the context of a multimodal interactive manipulation task involving successive local motion and handling commands. Our object exchange scenario was successfully run several times with different users.