Fully Asynchronous Neuromorphic Perception for Mobile Robot Dodging With Loihi Chips

摘要

Sparse and asynchronous sensing and processing in natural organisms lead to ultra low-latency and energy-efficient perception. Inspired by these characteristics, event cameras have attracted extensive attention from academia and industry. However, the mainstream event stream processing paradigms (e.g., event frames, 3D voxels) encounter issues such as feature loss, event stacking, and high computational burden. These problems deviate from the intended purpose of event cameras. To address these issues, we propose a fully asynchronous neuromorphic paradigm that integrates event cameras, spiking networks, and neuromorphic processors (Intel Loihi). This paradigm can faithfully process each event asynchronously as it arrives, mimicking the spike-driven signal processing in biological brains. We first propose a Key-Event-Point module to extract the key event stream from the raw event stream, effectively addressing the issue of limited transmission bandwidth when processing events on neuromorphic processors. Then, we propose the complete pipeline for the dodging network based on spiking neural networks to achieve offline training and online inference. Finally, we compare the proposed paradigm with event frames and 3D voxels processing paradigms in detail on the real mobile robot dodging task. Experimental results show that our scheme exhibits better robustness than image-like methods with different time windows and light conditions. Additionally, the energy consumption per inference of our scheme on the embedded Loihi processor is only 4.30% of that of the event spike tensor method on NVIDIA Jetson Orin NX with energy-saving mode, and 1.64% of that of the event frame method on the same neuromorphic processor. To the best of our knowledge, this is the first time that a fully asynchronous neuromorphic paradigm has been implemented for solving sequential tasks on a real mobile robot.<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i>—As a neuromorphic visual sensor, the event camera offers a novel approach to achieving robust, low-power perception in robotics, owing to its high temporal resolution, wide dynamic range, and low information redundancy. However, the sparse and asynchronous nature of event streams presents a significant challenge for data processing. The mainstream approach preprocesses event streams into various representations (e.g., event frames, 3D voxels) before performing subsequent operations, leading to issues such as feature loss, event stacking, and high computational burden. In this paper, we propose a fully neuromorphic system that leverages the asynchronous characteristics of event streams and spiking neural networks to enable asynchronous processing of events. Our asynchronous processing of event streams enhances robustness across different time windows and lighting conditions while significantly reducing power consumption during inference. Our fully asynchronous neuromorphic system has been validated on a real mobile robot and is expected to advance the robot perception system towards biological perception.