Exploring the Capabilities and Limitations of Large Language Models for Zero-Shot Human-Robot Interaction

Abstract

Human-robot interaction (HRI) is an evolving field with a growing emphasis on enabling robots to understand and perform tasks based on natural language commands. Recently, Large Language Models (LLMs) have emerged as a promising tool for such tasks, offering the potential to enable zero-shot learning and flexible interaction without task-specific training. In this paper, we explore the use of LLMs for zero-shot navigation and exploration tasks in robotic systems, specifically evaluating their performance with the PR2 Clearpath and Khepera IV robots in a simulated environment. Our findings demonstrate promising results, particularly in the LLM’s ability to exhibit exploratory behavior and iterative reasoning when faced with ambiguous or incomplete visual input. These capabilities suggest a strong potential for LLMs in human-robot interaction. However, challenges were also identified, such as difficulties with target recognition, object misidentification, hallucination of information, and issues with movement execution, highlighting the need for improvements in these areas for real-world applications.