Learning a natural-language to LTL executable semantic parser for\n grounded robotics

Abstract

Children acquire their native language with apparent ease by observing how\nlanguage is used in context and attempting to use it themselves. They do so\nwithout laborious annotations, negative examples, or even direct corrections.\nWe take a step toward robots that can do the same by training a grounded\nsemantic parser, which discovers latent linguistic representations that can be\nused for the execution of natural-language commands. In particular, we focus on\nthe difficult domain of commands with a temporal aspect, whose semantics we\ncapture with Linear Temporal Logic, LTL. Our parser is trained with pairs of\nsentences and executions as well as an executor. At training time, the parser\nhypothesizes a meaning representation for the input as a formula in LTL. Three\ncompeting pressures allow the parser to discover meaning from language. First,\nany hypothesized meaning for a sentence must be permissive enough to reflect\nall the annotated execution trajectories. Second, the executor -- a pretrained\nend-to-end LTL planner -- must find that the observe trajectories are likely\nexecutions of the meaning. Finally, a generator, which reconstructs the\noriginal input, encourages the model to find representations that conserve\nknowledge about the command. Together these ensure that the meaning is neither\ntoo general nor too specific. Our model generalizes well, being able to parse\nand execute both machine-generated and human-generated commands, with\nnear-equal accuracy, despite the fact that the human-generated sentences are\nmuch more varied and complex with an open lexicon. The approach presented here\nis not specific to LTL: it can be applied to any domain where sentence meanings\ncan be hypothesized and an executor can verify these meanings, thus opening the\ndoor to many applications for robotic agents.\n