VocaWatcher: Natural singing motion generator for a humanoid robot

Abstract

In this paper, we describe VocaWatcher, a novel robot motion generator that enables a humanoid robot to sing with realistic facial expressions and naturally synthesized singing voices. This robot singer is an important and attractive humanoid robot application for the entertainment scene; moreover, it promotes state-of-the-art integration of robot engineering, music processing, and image processing. To overcome the difficulties of generating natural facial expressions that are precisely synchronized with singing voices, VocaWatcher imitates a human singer by analyzing a video clip of a human singing, recorded by a single video camera. VocaWatcher can control mouth, eye, and neck motions by imitating the corresponding human movements, which are estimated without using any markers in the video. It can also synthesize singing voices by imitating the pitch and dynamics of the human singing in the same video.