Facial Emotion Recognition using Video Visual Transformer and Attention Dropping

Abstract

Understanding human emotions is a fundamental task in the affective computing community due to its wide range of applications including robotics, psychology, or computer science. Recognizing emotions is an important factor in human interaction, helping people to convey intentions, empathy and in some cases the actual meaning of a message. In this paper, we have introduced a novel discrete emotion recognition framework based on visual information analysis. At the technical level, the core of the proposed methodology involves a novel video-visual transformer extended with an attention dropping stage that allows extracting the spatiotemporal locations of the most relevant facial regions illustrating the peak of emotion. The experimental evaluation conducted on two publicly available datasets CREMA-D and RAVDESS validates the proposed methodology, which lead to average accuracy scores of 82.16% and 85.56%, respectively.