Hybrid lip shape feature extraction and recognition for human-machine interaction

摘要

Dumb and deaf people are unable to interact with robots using traditional voice-based human-machine interfaces (HMI). Lip motion is a useful way for these people to communicate with machines, even for normal people in extremely noisy environments. However, the recognition of lip motion is a difficult task since the region of interest (ROI) is non-linear and noisy. This paper proposes a novel lip shape feature extraction method to deal with the difficulty, based on hybrid dual-tree complex wavelet transform (DT-CWT) and discrete cosine transform (DCT). The approximate shift invariance of DT-CWT is utilised to make the same lip shape have the same feature vector when the lips are in different positions in the ROI. Then, DCT is used to extract coefficients from the feature vector generated by DT-CWT, and to choose the larger coefficients to obtain the key information of lip shape and reduce the dimensions of a feature vector. The experimental results show that this method can greatly improve the accuracy of lip shape recognition, and enhance the robustness of the lip shape-based HMI.