Skeleton‐Guided Action Recognition with Multistream 3D Convolutional Neural Network for Elderly‐Care Robot

Abstract

With the arrival of a global aging society, elderly‐care robots are becoming more and more attractive and can provide better caring services through action recognition. This article presents a skeleton‐guided action recognition framework with multistream 3D convolutional neural network. Two parallel dual‐stream lightweight networks are proposed to enhance the feature extraction ability of human action and meanwhile reduce computation. Two different modes of skeleton input video are constructed to improve the recognition accuracy by decision fusion. The backbone networks adopt Resnet‐18, the feature fusion layer and sliding window mechanism are both designed, and two cross‐entropy losses are used to supervise their training. A dataset (named elder care action recognition (EC‐AR)) with different categories of action is built. The experimental results on HMDB‐51 and EC‐AR datasets both demonstrate that the proposed framework outperforms the existing methods. The developed method is also applied to a prototype of elderly‐care robots, and the test results in home scenarios show that it still has high recognition accuracy and good real‐time performance.