FAFNs: Frequency-Aware LiDAR–Camera Fusion Networks for 3-D Object Detection

Abstract

The 3-D object detection is a crucial task for autonomous driving and robotics. However, it faces various challenges attributed to the complexity and sparsity of 3-D data. Light detection and ranging (LiDAR) and camera are two common sensors that provide complementary information for 3-D object detection. LiDAR provides accurate depth and shape information, while camera captures rich semantic and texture information. However, most existing methods for LiDAR–camera fusion either ignore the frequency-domain information or use simple concatenation or elementwise operations to fuse the features, which may not fully realize the potential of both modalities. This article proposes frequency-aware LiDAR–camera fusion networks (FAFNs) that leverage the frequency-domain information to enhance the LiDAR and camera feature maps. Specifically, we design a frequency-aware feature enhancement module (FAFEM) that adaptively learns the frequency importance of each modality and assigns different weights to different frequency bands. We also design a frequency-aware multiscale query initialization strategy that enriches the object query with frequencywise information and multiscale variations. We conduct the evaluation of our FAFNs using two well-established benchmarks for 3-D object detection, nuScenes and Waymo, which demonstrates its competitive performance over existing state-of-the-art methods.