HUGGA: Human-like Grasp Generation with Gripper’s Approach State Using Deep Learning

Abstract

In this study, an attempt has been made to develop a new method for producing a successful grasp based on human behavior, by considering the gripper’s approach state in the final grasp of an object. The dataset collected for the purpose of this paper consists of 14 different daily usage objects with different orientations (In total 2526 images) which a human approaches and grasps them. Data collection has been done based on a protocol shown in TableI to cover all achievable states. From Fig1, it can be seen that the dataset includes one image from a single object without the user’s hand. The second image contains the human hand approach and its configuration regarding the object which is subject to be grasped and the third image represents the object which is grasped by the human hand and thus evidently contains both object and human hand grasp configuration. It is important to acknowledge that providing a static hand image does not suffice to facilitate the acquisition of human-like grasping capabilities by robot grippers, so, a mapping from hand fingers to a two-finger gripper should be implemented to fill this gap. In the collected dataset, all grasps are performed by the index finger and thumb to make it possible to map the coordinates of each finger on a gripper finger. Using MediaPipe, which operates a single shot detection network, with only an RGB image containing a human hand the index finger and thumb landmarks were extracted. Finally, a detailed human approach and grasp information were available, and with the collected data, a two-input and one-output deep neural network was trained. The designed model produced a grasp with a human pattern by receiving only the RGB image of the object and the approach vector to the object. Ultimately, through testing and evaluation based on existing criteria, the trained network successfully produced a grasp with 61% accuracy in grasp rectangle overlap, 97% in grasp angle and overall 60.5% met two established grasp criteria.