Harvesting tomatoes with a Robot: an evaluation of Computer-Vision capabilities

Abstract

This paper presents the practical application and comparison of two 3D cameras: Intel RealSense D435 and Zivid Two used for tomato detection and localization in laboratory and greenhouse conditions. The performance of the cameras with YOLO model was studied and evaluated based on the image quality, precision of detection, accuracy of 3D coordinates of the detected tomatoes and the price of the devices. The proposed prototype with Universal Robot UR5 can detect and harvest tomatoes both in the laboratory and greenhouse. We cover three main findings as the results of our research. First, apart from cost, Zivid Two scores significantly better results than RealSense on all other aspects of our evaluation. Accuracy and precision obtained with Zivid are much higher than with Intel camera. For tests in the greenhouse, RealSense even becomes unusable due to the specific lighting. Second, we achieve the state-of-the-art average detection time of 2.1 s. due to the lightweight detection execution with YOLO. Furthermore, all tomatoes seen in the camera frame are recognized within the execution time and all detected tomatoes are true positives, what provides us consequent time saving in scheduling the harvesting tasks. Third, and main novelty of this research, with an overall harvesting time of 16 s., we reach beyond the state-of-harvesting time of 33 s. with only increasing the speed of movement of our robotic arm UR5. This paper was written based on the tomato harvesting prototype being developed under Satobotti project (EU ERDF fund - A77791).