Answering correctly or incorrectly: the impact of multimodal feedback from education robot voice gender and body movements on users’ learning outcomes

Abstract

Despite the growing interest in multimodal communication in human-robot interaction (HRI), research about how to design robots' multimodal feedback in educational contexts remains limited. This study employed a 2 × 3 (voice gender: male vs female × body movements: head vs hand vs combined movements) within-subject experimental design to investigate the effects of multimodal feedback (voice and body movements) on learning outcomes from three perspectives: affective outcomes, visual attention, and cerebral activity. The results revealed that voice gender and body movements significantly influenced users' affective outcomes and cerebral activity. Voice gender impacted visual attention, and an interaction effect between voice gender and body movements was observed in cerebral activity. Moreover, this research suggested that robots with female voices and combined movements promoted learning outcomes and revealed that robot voice gender and body movements may independently shape users' learning outcomes in low-demand cognitive feedback.