Do You Understand Me, DUM-E? Enabling Seamless Human-Robot Communication Through Augmented Reality

Abstract

This paper introduces an Augmented Reality (AR) system that facilitates seamless intent communication between a human and a collaborative robot during complex assembly tasks. This system leverages a Unity-ROS pipeline to integrate AR with multimodal input and output capabilities, providing a rich, interactive interface for Human-Robot Interaction (HRI). To evaluate the system’s effectiveness, a series of user studies were conducted with participants performing assembly tasks under different communication scenarios involving different interaction modes and intent interpretation feedback. The effects of the AR system on participants’ task performance and user satisfaction were analyzed, indicating that the AR-enabled multimodal interface significantly improves communication clarity, reduces errors, and enhances overall task efficiency compared to traditional methods with no AR feedback. Immediate visual feedback significantly reduces task completion times and error rates. Higher levels of user confidence and trust in the robot were observed when visual feedback was provided, highlighting the importance of transparency in robot actions.