Domain centralization and cross-modal reinforcement learning for vision-based robotic manipulation

Abstract

Vision-based robotic manipulation with deep learning method has achieved substantial advances in the field of automatic agriculture, which can be deployed and applied in the picking, sorting and transporting of agricultural products and so on. Deep reinforcement learning (DRL) is one of the learning-methods that help the robot learn the policy itself by exploration and exploitation. Training real robots with DRL would take a great price that limits its application scope. Some approaches train the DRL policy in simulation and deploy the model to real robot by transferring the images in simulator to that of the real world. However, this method requires pre-collected images as training data for each real scene. In this paper, a domain centralized approach is proposed as the sim-to-real perception module to capture the task-specific characteristics of the vision regardless of the reality gap between simulation and the real environments. Another challenge for vision-based manipulation is the learning difficulty caused by the high-dimensional vision input. Here we propose a cross-modal reinforcement learning scheme by leveraging the full system states to provide additional guidance. The experimental results show that the proposed method can perform a real robot grasping task without real-world data and outperforms current methods with the same experimental settings. Keywords: sim-to-real, robotic manipulation, agricultural application, domain centralization, cross-modal learning DOI:  10.33440/j.ijpaa.20200302.77  Citation: Yang K, Zhang Z P, Cheng H, Wu H D, Guo Z Y. Domain centralization and cross-model reinforcement learning for vision-based robotic manipulation.  Int J Precis Agric Aviat, 2020; 3(2): 48–55.