Transfer-Learning-Based Gesture and Pose Recognition System for Human–Robot Interaction: An Internet of Things Application

Abstract

Human–machine interactions have become increasingly crucial in the current era of the Internet of Things (IoT). Mutual feedback is critical for adjusting machine operations to improve the efficiency of human–machine interactions. Imaging can be easily conducted in various contexts to acquire large volumes of visual information, such as that regarding human gestures. In the present study, machine learning technology, which is the driving technology for intelligent processing in IoT systems, was adopted to develop a system for identifying and classifying six hand gestures and five body poses. Gesture and pose data were collected and analyzed using multiple algorithms to construct classification models for pose recognition. Data for one individual were used to train base gesture and pose recognition models, and transfer learning was then performed to adapt these base models to the gesture and pose data of other individuals. The adapted models achieved satisfactory recognition accuracy. The developed gesture and pose recognition models were tested by employing them to control a robotic arm and an automated guided vehicle, respectively. All models achieved accuracy rates of >97%, thereby confirming the effectiveness of the proposed machine-learning-based method for gesture and pose recognition.