Vision-Based Control of Construction Robots Using Transfer Learning Approach

Abstract

This study explores the potential of vision-based control of robots for construction applications using a transfer learning approach. Repetitive and arduous construction tasks cause musculoskeletal disorders in human workers which can be performed safely by robots. Unlike manufacturing processes, construction tasks are complex. Collaborative robots utilize human intelligence to adapt to the complexities of the construction site. Interaction between humans and robots is a challenge because of the limited infrastructure and computational training of construction workers. Vision-based hand gesture control is implemented using transfer learning by using a pre-trained hand detection model called Mediapipe Hands and using its output as features to train another artificial neural net (ANN) to classify different hand gestures. After optimizing the model parameters, 99.43% training and 99.11% validation accuracies were achieved. The results of validation performed through experiments in a simulated environment show the potential of using hand gestures for the control of construction robots.