Multi-modal Emotion Recognition

Abstract

Emotion recognition is crucial in fields like human-computer interaction, mental health, and social robotics, where understanding and responding to emotions is key to effective communication. Recent research has shown that combining multiple types of data, such as visual and audio signals, can improve the accuracy of emotion recognition systems. In this study, we present a multi-modal emotion recognition model that considers both visual and audio features to enhance emotion classification. The model extracts facial features from video frames using OpenFace and EfficientNet, and utilises the Mel-frequency cepstral coefficients (MFCC) and other audio features extracted by Wav2Vec 2 from soundtracks. The model uses a transformer-based feature analyzer to capture temporal dependencies and interactions between the modalities. Finally, the combined feature representation is fed into a classifier that predicts the emotional state. This approach allows for a more comprehensive understanding of emotions by leveraging both visual and auditory cues, leading to improved accuracy in emotion recognition tasks. Experiments show that the proposed model performs better than existing models on the RAVDESS and SAVEE datasets.