Unlocking the power of multi‐modal fusion in 3D object tracking

Abstract

Abstract 3D Single Object Tracking plays a vital role in autonomous driving and robotics, yet traditional approaches have predominantly focused on using pure LiDAR‐based point cloud data, often neglecting the benefits of integrating image modalities. To address this gap, we propose a novel Multi‐modal Image‐LiDAR Tracker (MILT) designed to overcome the limitations of single‐modality methods by effectively combining RGB and point cloud data. Our key contribution is a dual‐branch architecture that separately extracts geometric features from LiDAR and texture features from images. These features are then fused in a BEV perspective to achieve a comprehensive representation of the tracked object. A significant innovation in our approach is the Image‐to‐LiDAR Adapter module, which transfers the rich feature representation capabilities of the image modality to the 3D tracking task, and the BEV‐Fusion module, which facilitates the interactive fusion of geometry and texture features. By validating MILT on public datasets, we demonstrate substantial performance improvements over traditional methods, effectively showcasing the advantages of our multi‐modal fusion strategy. This work advances the state‐of‐the‐art in SOT by integrating complementary information from RGB and LiDAR modalities, resulting in enhanced tracking accuracy and robustness.