In-Hand Object Classification and Pose Estimation With Sim-to-Real Tactile Transfer for Robotic Manipulation

Abstract

Dexterous robotic grasping gains great benefits from tactile sensation, but delicate object exploration by tactile information is challenged by difficulty in rich and efficient data production. In this letter, we propose a tactile-based in-hand object perception approach, empowered by a sim-to-real approach toward a data-efficient learning process. A high-fidelity tactile input, measured by a pair of vision-based tactile sensors, was represented as a point cloud facilitating dual functionality of object classification and the associated pose estimation. For the classification, we constructed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PoinTacNet</i> , a variation of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PointNet</i> to fit into tactile data processing. A reliable simulation methodology on tactile input was employed for the pretraining of the model, transferred to the fine-tuning process facing real tactile data. Taking inspiration from human behaviors, a re-grasping strategy was imparted by means of conditional accumulation of class likelihood distribution. The result of the framework facilitates a high object classification accuracy of 83.89 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> on the ten objects from McMaster-Carr's CAD models, which is significantly improved by the re-grasping. In addition, a set of benchmarks displays the computational efficiency in the sim-to-real transfer. In line with the successful classification, the posture of in-hand objects is estimated using point cloud registration algorithms, achieving an average angular and translational RMSE of 5.03 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^\circ$</tex-math></inline-formula> and 2.41 mm, respectively. The proposed approach has the potential to enable robots to attain human-like haptic exploration skills for perceiving unstructured environments.