Closing the Simulation-to-Reality Gap using Generative Neural Networks: Training Object Detectors for Soccer Robotics in Simulation as a Case Study

Abstract

In order to address the simulation-to-reality-gap, in this paper a methodology for the real-time generation of realistic images in robotic simulation environments is proposed. The images rendered by the simulator are first segmented, and then a generative neural network transforms them into realistic images. This allows training object recognition methods in situations dynamically generated by the simulator, but using realistic images. The generative neural network is trained using a database obtained using an instance segmentation network (Mask R-CNN). The whole methodology is validated in the soccer robotics domain. The reported experiments show that CNN based object detectors trained in simulation, using the generated realistic images, can be directly transferred to reality, where state-of-the art results are obtained. Moreover, we show how the training process of these detectors is fast, easy, and does not require the repetitive use of robots, which is time consuming for humans.