Leveraging Deep Learning and RGB-D Cameras for Cooperative Apple-Picking Robot Arms

摘要

<sc>Abstract.</sc> We report on a novel method of using two cooperating robot arms to pick apples in a typical unstructured orchard. Each robot arm is equipped with an Intel Realsense D435 RGB-D (color plus depth) camera at its wrist. The first robot arm (termed the “Search arm”) is used to detect apples and also to survey a given orchard tree's volumetric space for clear paths. The second, “Grasp arm”, is positioned relatively closer to the tree and is designated with approaching fruit with the intent to harvest. A custom-trained deep learning-based object detection algorithm called YOLO (You Only Look Once) is used for finding apples in the cluttered scene. Apple location and clear path information is encoded into a graph and used for planning. The arms are driven by a finite state machine with information provided by the graph. Computer simulation and real world experimental results are reported. Based on our results to date, solutions are proposed to improve robustness, apple localization, and to minimize average time to pick all feasible apples.