Autonomous Vision-based Robotic Grasping of Household Objects: A Practical Case Study

Abstract

Autonomous robotic grasping has emerged as a valuable capability for automating everyday household tasks like table setting (bossing), cleaning, and meal preparation. Still, accurately detecting and manipulating diverse everyday objects in unstructured home environments remains a significant challenge. In this study, an integrated robotic system combining YOLOv8s-seg deep learning for object detection and segmentation with computer vision techniques to determine optimal grasping points is proposed. The accuracy of this model has also been compared with other models through their mAP0.5, mAP.05-.95, precision and recall. A custom dataset of over 1000 images containing common household objects was created, with manual annotation and augmentation to train the YOLOv8s-seg segmentation model. The approach processes input images to understand desired arrangements and identify target objects and their locations for grasping. The detected objects are localized using camera-to-robot calibration and aligned with estimated orientations from computer vision techniques. Extensive tests have proven that representative household objects in cluttered arrangements have a grasping success rate of over 90% using a 3DOF Delta Parallel robot with a 2-finger gripper guided entirely by visual perception. By overcoming real-world challenges like transparent and reflective materials, the perception-guided robots provide precise and efficient robotic grasping in unstructured home environments. This integrated approach combines robust perception and localization, providing promising evidence for reliable automation of repetitive household organization tasks<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>.<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>A video of real-time grasp performance is uploaded as supplementary material.