Photonics Neural Networks for Multimodal Recognition Based on the Self-Activated MAC Function of DFB-SA

Abstract

Inspired by biological nervous systems, multimodal deep learning integrates multimodal information to enhance perception and decision-making, yet its high computational demands challenge traditional microelectronic processors in energy efficiency and speed. Photonic neuromorphic computing offers a promising solution, but implementing linear weighting and nonlinear activation typically requires different photonic materials and devices, complicating large-scale integration. Here, we propose and demonstrate a hybrid optoelectronic neural network architecture based on a distributed feedback laser with a saturable absorber (DFB-SA) array, designed to mimic biological audiovisual fusion for multimodal recognition tasks. This architecture leverages the self-activated multiply accumulate (MAC) function of the DFB-SA laser, seamlessly integrating both linear weighting and nonlinear activation into a single device, thus significantly improving integration efficiency. The proposed multimodal neural network outperforms unimodal recognition methods in recognition accuracy and robustness under challenging conditions, achieving over 90% accuracy in hardware inference. Each computational core achieves a speed of 1.6 GOPS, an energy efficiency of 38.1 GOPS/W, and a unit area speed of 21.3 GOPS/mm2, with overall performance scaling linearly with the number of cores. Furthermore, we develop a robot obstacle avoidance system utilizing the self-activated MAC function of DFB-SA laser neurons. This work presents a high-performance computing hardware platform for multimodal deep learning, demonstrating its potential for simulating biological multisensory recognition and enabling scalable photonic AI models.