Novel Skeleton-based Action Recognition Using Covariance Descriptors on Most Informative Joints

Abstract

Human action recognition has attracted much attention of research community in recent years due to its large application domain such as human-robot interaction, gaming, surveillance, etc. Action recognition can be implemented in either single-modal (color, depth, skeletal) or multi-modal schemes. Our research focuses on human action recognition based on skeletal information. In literature, a Temporal Hierarchy of Covariance Descriptors on 3D Joints (Cov3DJ) was proposed to exploit information from time dimension of skeletal data. Since each joint has a certain level of engagement in an action, another approach aims at selecting joints which are most informative for action recognition. In this paper, a novel framework, named as Covariance Descriptor on Most Informative Joints (CovMIJ), is proposed to benefit from the simplicity of representation via covariance descriptor and noise immunity by using only Most Informative Joints (MIJ) for action recognition. Extensive experiments between CovMIJ, Cov3DJ and state-of-the-art Res-TCN (Temporal Convolutional Neural Networks with Residual Units) on two public datasets (MSR-Action3D and CMDFALL) show the efficiency of our proposal. On MSR-Action3D dataset, accuracy of CovMIJ achieves 93.6% while that of Cov3DJ is only 90.53%. Improvement is found on CMDFALL dataset with accuracy of 64.72% for CovMIJ compared to 61.34% for Cov3DJ. On CMDFALL dataset, CovMIJ with F1 score of 62.5% outperforms the deep learning network Res-TCN with F1 score of 39.38%.