Integrated visual representations for programming with real-world input and output

Abstract

As computers become more pervasive, more programs deal with real-world input and output (real-world I/O) such as processing camera images and controlling robots. The real-world I/O usually contains complex data hardly represented by text or symbols, while most of the current integrated development environments (IDEs) are equipped with text-based editors and debuggers. My thesis investigates how visual representations of the real world can be integrated within the text-based development environment to enhance the programming experience. In particular, we have designed and implemented IDEs for three scenarios, all of which make use of photos and videos representing the real world. Based on these experiences, we discuss "programming with example data," a technique where the programmer demonstrates examples to the IDE and writes text-based code with support of the examples.