Lensless Sensing of Facial Expression by Transforming Spectral Attention Features

Abstract

Camera-based facial expression recognition (FER) systems have made significant progress in many areas, such as safe driving and robotics, but they also pose challenges in terms of the imaging system size and privacy-preserving. Emerging lensless cameras are distinguished by their small size and visual privacy due to diffused measurements. Existing lensless FER methods first reconstruct images from lensless measurements and then perform the FER task on reconstructed images. However, these reconstructed images still contain some privacy-sensitive information, which still suffers from privacy leakage. In this paper, we propose an end-to-end network called LenslessFET to predict facial expressions directly from lensless measurements without image reconstruction, thus inheriting the privacy-preserving merits of lensless cameras. To this end, we propose the Spectral Attention module, which learns adaptive filters to extract expression information in the frequency domain. Besides, we observe that spectral attention features contain some undesirable noises that hinder expression recognition. To address the problem of noise interference in spectral attention features, we group them according to their noise level and apply the Basis Modulation Transformer to enhance expression information from these noisy features. Extensive experiments show that LenslessFET achieves state-of-the-art performance on the real-captured dataset, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e</i> ., FCFD dataset, and simulated FER datasets, <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e</sup> ., RAF-DB <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">+</sup> and FERPlus <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">+</sup> . Our code will be available at this link.