ATTSF-Net: Attention-Based Similarity Fusion Network for Audio-Visual Emotion Recognition

Abstract

Emotional factors play a pivotal role in fields such as autonomous driving and intelligent emotional robotics. The accurate extraction of emotional factors is instrumental in reducing error rates within these domains. With the continuous deepening of exploration in the emotional domain, rich multimodal data has progressively supplanted unimodal data. Nevertheless, current multimodal approaches still grapple with the following challenges: 1.Partial loss of information both within individual modalities and across different modalities. 2.Incorrect extraction of modality-invariant features. To facilitate multimodal interaction and address the aforementioned issues, this paper proposes an Attention-based Similarity Fusion Network (ATTSF-Net) for audio-visual emotion recognition. The network is based on multimodal data and comprises the proposed Cross-Multimodal Block (CMB), Similarity Adjustment Block (SAB), and Audio-Visual Auxiliary Modules (AVAM). CMB employs a cross-modal attention mechanism and model-level fusion to facilitate interactions between modalities. SAB is designed to learn modality-invariant features. AVAM utilizes additional audio-visual auxiliary networks to provide supplementary emotional information, enabling the full extraction of intra-modal information. A similarity loss function based on Kullback-Leibler (KL) divergence is designed to ensure the consistency of the learned audio-visual emotional information. The proposed model achieves an accuracy of 88.67% on the RAVDESS dataset (8.67% higher than human) and an unweighted accuracy (UA) of 81.93% and a weighted accuracy (WA) of 79.77% on the IEMOCAP dataset (4.01% and 5.86% higher than transfer learning, respectively). A series of visualization experiments are conducted to demonstrate the effectiveness of the proposed model.