Vim-Grasp: A Mamba-Based Gesture Generation Network for Multi-Scale Object Grasping

Abstract

The importance of robotic grasping technology in the industrial and service sectors cannot be overstated. However, existing visual grasping methods still face challenges in terms of robustness and accuracy in complex environments. Therefore, it is crucial to investigate efficient and accurate visual grasping methods. The current visual grasping methods are mainly based on CNN-based model architectures, and a few methods are also based on the Transformer. For the current visual grasping methods, generally speaking, there are several major problems: first, the robustness of the existing methods is insufficient in complex real-world environments, such as lighting changes and diverse object shapes. Second, for applications that require high-precision localization and manipulation, there is still room for improvement in the accuracy of existing methods. Finally, these methods may require a large amount of computational resources, which is not favorable for real-time applications or mobile robot applications. In order to solve the above problems, this thesis proposes a network model called Vim-Grasp, which is specialized for visual grasping tasks. The model employs several approaches to improve efficiency and accuracy: the first approach is the introduction of the mamba model, which is first applied to visual grasping tasks, and effectively compresses the visual representation through a bidirectional selective state space, which improves the memorization ability during long sequential inputs. The second approach uses a jump-connected hierarchical encoder-decoder architecture to facilitate multi-scale feature fusion, passing shallow features from the encoder to the decoder to achieve global information fusion and enhance feature learning capability. The third approach uses a lightweight design to optimize the model structure so that only one forward propagation is needed to output the grasping pose and improve computational efficiency. Experimental results on Cornell and Jacquard datasets show that our Vim-Grasp model achieves 98.23% and 95.4% accuracy and is well suited for visual grasping tasks.