VTG: A Visual-Tactile Dataset for Three-Finger Grasp

Abstract

Three-fingered hands can offer more contact points and flexible fingertip configurations, enabling complex grasping modes and finer manipulations for objects of various shapes and sizes. However, existing research on visual-tactile integrated robotic grasping primarily focuses on grippers, and lacks a general dataset merging visual and tactile information for the entire grasping process. In this letter, we introduce the VTG dataset, which can be used for various aspects of three-fingered robotic grasping control. The VTG dataset includes three-view point clouds of objects, grasping modes, and finger angles of three-fingered hands, as well as tactile data at multiple spatial contact locations during the grasping process. By integrating visual and tactile information, we develop a robotic grasping controller that leverages a grasping stability prediction module and a grasping adjustment module. By representing tactile data as a static graphical structure based on the spatial distribution of tactile sensors, the grasping stability prediction module is constructed based on a multi-scale graph neural network, MS-GCN. It combines multi-scale graph topological features with various grasping modes, and achieving an accuracy of 98.4% in robotic grasping stability prediction. Additionally, this controller successfully adapts to unknown objects of varying hardness and shapes, providing stable grasping within approximately 0.4 s after contact.