A Neuromorphic Vision and Feedback Sensor Fusion Based on Spiking Neural Networks for Real‐Time Robot Adaption

Abstract

For some years now, the locomotion mechanisms used by vertebrate animals have been a major inspiration for the improvement of robotic systems. These mechanisms range from adapting their movements to move through the environment to the ability to chase prey, all thanks to senses such as sight, hearing, and touch. Neuromorphic engineering is inspired by brain problem‐solving techniques with the goal of implementing models that take advantage of the characteristics of biological neural systems. While this is a well‐defined and explored area in this field, there is no previous work that fuses analog and neuromorphic sensors to control and modify robotic behavior in real time. Herein, a system is presented based on spiking neural networks implemented on the SpiNNaker hardware platform that receives information from both analog (force‐sensing resistor) and digital (neuromorphic retina) sensors and is able to adapt the speed and orientation of a hexapod robot depending on the stability of the terrain where it is located and the position of the target. These sensors are used to modify the behavior of different spiking central pattern generators, which in turn will adapt the speed and orientation of the robotic platform, all in real time. In particular, experiments show that the network is capable of correctly adapting to the stimuli received from the sensors, modifying the speed and heading of the robotic platform.