A framework for recognizing and executing verb phrases

Abstract

Today, the physical capabilities of robots enable them to perform a wide variety of useful tasks for humans, making the need for simple and intuitive interaction between humans and robots readily apparent. Taking natural language as a key element of this interaction, we present a novel framework that enables robots to learn qualitative models of the semantics of an important class of verb phrases, such as “follow me to the kitchen,” and leverage these verb models to perform two tasks: Executing verb phrase commands, and recognizing when another agent has performed a given verb. This framework is based on a qualitative, relational model of verb semantics called the Verb Finite State Machine, or VFSM. We describe the VFSM in detail, motivating its design and providing a characterization of the class of verbs it can represent. The VFSM supports the recognition task natively, and we show how to combine it with modern planning techniques to support verb execution in complex environments. Grounded natural language semantics must be learned through interaction with humans, so we describe methods from learning VFSM verb models through natural interaction with a human teacher in the apprenticeship learning paradigm. To demonstrate the efficacy of our framework, we present empirical results showing rapid learning and high performance on both the recognition and execution tasks. In these experiments, the VFSM is able to consistently outperform a baseline method based on recent work in the verb learning literature. We close with a discussion of some of the current limitations of the framework, and a roadmap for future work in this area.