A vision system based on CNN-LSTM for robotic citrus sorting

Abstract

Compared with manual sorting of citrus fruit, vision-based sorting solutions can help achieve higher accuracy and efficiency. In this study, we present a vision system based on CNN-LSTM, which can cooperate with robotic grippers for real-time sorting and is readily applicable to various citrus processing plants. A CNN-based detector was adopted to detect the defective oranges in view and temporarily classify them into corresponding types, and an LSTM-based predictor was used to predict the position of the oranges in a future frame based on image sequential data. The fusion of CNN and LSTM networks enabled the system to track defective ones during rotation and identify their true types, and their future path was also predicted which is vital for predictive control of visually guided robotic grasping. High detection accuracy of 94.1% was obtained based on experimental results, and the error for path prediction was within 4.33 pixels 40 frames later. The average time to process a frame was between 28 and 62 frames per second, which also satisfied real-time performance. The results proved the potential of the proposed system for automated citrus sorting with good precision and efficiency, and it can be readily extended to other fruit crops featuring high versatility.