Robust finger gesture recognition on handheld devices for spatial programming of industrial robots

Abstract

We have developed a spatial programming system for industrial robots based on gestures and Augmented Reality. In this respect, we aim for a markerless and robust finger gesture recognition on the handheld devices. The user should be enabled to interact with virtual objects representing the robot program in front of the device. The great challenge is to be fast and robust in terms of diffuse background and changing environmental conditions. In this contribution, we present an image-based finger gesture recognition approach which is tailored to efficient implementation on mobile devices, e.g. Smartphones and Tablet PCs. For this purpose, we introduce an adaptive, rule-based skin classifier. The tracking is done by a specific particle filter: the Condensation algorithm. Due to further implemented improvements, we reduce the number of particles in order to reduce the computational time of the algorithm. The gesture recognition is implemented in an Android app, enabling the user to translate and rotate virtual object within an Augmented Reality application in real-time.