Cross-media translation based on mental image directed semantic theory toward more comprehensible multimedia communication

Abstract

Computerized intelligent systems should be more human-friendly considering rapid increase of aged societies, floods of multimedia information over the WWW and development of robots for practical use. Even for people without any physical handicap, it is not always easy to communicate with each other comprehensively by limited information media. Not only for the cases between humans but also for humans and robots, employment of another information medium is very helpful and therefore cross-media translation is very important during such a media-limited communication. Mental image directed semantic theory (MIDST) has proposed a cognitively motivated methodology for formalizing spatio-temporal relationships conceptualized in natural language. This paper presents the method and the experiment of cross-media translation based on MIDST, where natural language texts about static positional relations of physical objects are systematically interpreted into 2D pictures and vice versa.