A Novel Emotion Recognition System for Human–Robot Interaction (HRI) Using Deep Ensemble Classification

Abstract

Human emotion recognition (HER) has rapidly advanced, with applications in intelligent customer service, adaptive system training, human–robot interaction (HRI), and mental health monitoring. HER’s primary goal is to accurately recognize and classify emotions from digital inputs. Emotion recognition (ER) and feature extraction have long been core elements of HER, with deep neural networks (DNNs), particularly convolutional neural networks (CNNs), playing a critical role due to their superior visual feature extraction capabilities. This study proposes improving HER by integrating EfficientNet with transfer learning (TL) to train CNNs. Initially, an efficient R‐CNN accurately recognizes faces in online and offline videos. The ensemble classification model is trained by combining features from four CNN models using feature pooling. The novel VGG‐19 block is used to enhance the Faster R‐CNN learning block, boosting face recognition efficiency and accuracy. The model benefits from fully connected mean pooling, dense pooling, and global dropout layers, solving the evanescent gradient issue. Tested on CK+, FER‐2013, and the custom novel HER dataset (HERD), the approach shows significant accuracy improvements, reaching 89.23% (CK+), 94.36% (FER‐2013), and 97.01% (HERD), proving its robustness and effectiveness.