A distributed and scalable person tracking system for robotic visual servoing with 8 dof in virtual reality TV studio automation

Abstract

In this paper, a distributed and scalable person tracking system for visual servoing using Industrial Robot Arms for virtual reality television studios (VR-TV) is presented. The system robustly tracks the moderator while freely moving, sitting or walking around the studio, and the estimation result can be used to drive the main broadcasting camera mounted on a large robotic arm connected with a pan tilt unit. The system consists of a person tracking system operating on the TV camera, which localizes the moderator in its field of view, and a overhead tracking system which localizes the moderator over the complete studio environment. The system is completely scalable in scenarios where a single scene is to be shot from multiple angles using multiple TV cameras. A common overhead tracking system monitors the moderator over the complete studio environment allowing the individual camera systems to initialize themselves and re-initialize in case of target loss. Application of the proposed tracking system to real-time VR-TV results in a robot cameraman, able to keep the moderator inside the screen with jitter-free viewpoint adjustments, as required by the VR scene rendering engine.