Soft Sensorized Physical Twin Enabling lab2field Transfer and Learning from Demonstration for Raspberry Harvesting

Abstract

Abstract Robotic harvesting is challenging manipulation task as it requires dexterity and robustness to handle delicate crops that can show complex structure and variable form. A compounding challenge is that the research and development methodology currently relies upon extensive field testing, which is inefficient and can also only happen during the harvesting season. With an urgent need to develop robotic solutions for harvesting to avoid crops being left in the fields unpicked, we explore how the research methodology for harvesting robots can be accelerated by learning the required interactions with delicate crops through human demonstration. Specifically, we focus on raspberry harvesting, a fruit which is challenging to harvest due to its fragility, and also which has a very narrow harvesting window limiting field trials. We propose leveraging soft robotic technologies to create a physical twin of the harvesting environment. This twin is a sensorized physical and visual simulator of the real raspberry plant with tuneable mechanical properties. This physical twin can be used to develop and optimize the harvesting robot and associated controller through human demonstrations. Furthermore, the robot can autonomously harvest the sensorized physical twin repetitively and use its sensor feedback to update and optimize its control parameters to best imitate a human harvester. We hypothesize, that by closing the reality gap between our physical twin and real world raspberry plants, the ability to achieve direct lab2field transfer increases, such that zero or minimal trials and adaptations in the field area is required. When the fully lab trained robot was tested in the field without any modifications, a 80\% successful harvesting success rate was achieved. We conclude the robot performs successfully under conditions simulated by the physical twin, while it is limited by unmodeled environmental conditions. Using this approach we demonstrate a new methodology for robotic harvesting research.