Vision-driven Collaborative Mobile Robotic Human Assistant System for Daily Living Activities

Abstract

Assistive robotics is a rapidly growing research area. They are increasingly in demand in industry, medical services, or even domestic environments. Many assistance systems were developed in the last decades to help with industry tasks mainly. However, the majority of assistive robotic systems can only execute commands according to pre-defined instructions and are not capable of social interaction with humans. To this end, a human-centered assistive robotic system that can automatically assist or completely help humans to accomplish their intentions is expected. In this paper, we first proposed a novel triplet concept that describes the intended task of humans but will be executed by the assistive robot. The triplet includes target object, subject and action enabling the robot to identify auxiliary subjects, desired objects and perform required actions. Furthermore, a vision-driven mobile robotic human assistant system for assisting humans in daily life tasks is developed. The proposed triplet was proven to be able to complete a fully autonomous robotic system as a predefined input guide. Sensor calibration, deep learning based object detection, manipulation, and grasping techniques are used to achieve this goal. We evaluated the proposed system using various simulation and experimental scenarios and confirmed its effectiveness.