Manual assembly action segmentation method based on spatiotemporal features

Abstract

Accurate segmentation of manual assembly action in an uncertain assembly scene is the premise and important foundation of robot autonomous learning to obtain action sequence. Therefore, this paper proposes a method of hand assembly action segmentation based on spatiotemporal features. This method takes the RGB-D video in the process of manual assembly demonstration as the research object. Firstly, the assembly scene graph of each video frame is constructed. On this basis, the spatial features of each video frame are extracted by using the graph network model. Then, the multi-stage temporal convolution network is used to process the spatial features in the time dimension to obtain the spatial and temporal features of each video frame. The spatiotemporal features pass through the softmax layer to obtain the recognition results of each frame of video, and the adjacent frames with the same action type are combined to obtain the manual assembly action sequence. This method obtains the temporal relationship between the front and back actions and avoids the problems of unaligned action boundaries and unsmooth action fragments in the hand assembly action segmentation method based on a single spatial feature. The experimental results show that the action editing score is improved from 78.18% to 99.28%, which verifies the effectiveness of the method.