From Visual to Multimodal Programming: Designing an Interface to Externalize Decomposition Thinking for Novice Learners

Abstract

Decomposition, the process of breaking down complex problems into manageable parts, is a fundamental component of computational thinking (CT) but remains challenging for novice learners. We present Spark, a multimodal programming interface that supports the externalization of decomposition thinking by organizing user-articulated goals into structured steps and enacting them through a tangible robot, making decomposition visible and open to inspection. The design of Spark is theory-driven, with its user interface aligned to three decomposition rationales: substantive, relational, and functional decomposition. In a study with 20 adult novices, we compared Spark with Scratch, an educational block-based visual programming interface. While both systems were associated with improvements in participants’ self-reported decomposition skills, only Spark was associated with measurable gains on objective assessments and significantly higher task success when experienced first, while maintaining comparable workload and completion times. Participants reported that externalizing the otherwise hidden process of decomposition made programming more tangible and motivating. These findings, highlighting the complementary roles of multimodal interaction in shaping novices’ decomposition experiences, inform the design of interactive programming interfaces that aim to support the externalization of reasoning processes. More broadly, our work contributes to the field of human-AI interaction in learning, illustrating how multimodal interaction can be responsibly integrated to scaffold reasoning, support reflection, and promote equitable participation in computing.