Deep Sandscapes: Design Tool for Robotic Sand-Shaping with GAN-Based Heightmap Predictions

Abstract

The aim of this research is to develop an adaptive and interactive design workflow for robotic sand-shaping. One of the challenges of working with natural materials is the increased level of complexity due to uncertain material behaviour. In this work, a generative adversarial network (GAN) is used to expedite material simulations. We train an image-to-image GAN to learn the relationship between planned excavation trajectories in existing sand states (input) and the modified sand states after excavation (ground truth). Data is collected prior to learning by an autonomous excavation routine. This routine (1) generates random trajectories that are executed by a robotic-arm; (2) uses an RGB-D camera to capture sand states as heightmaps before and after robotic interaction. Our GAN-assisted design tool predicts rearranged sandscapes by providing robotic excavation trajectories. Integrated into CAD software, an interactive and iterative design environment is realised for robotic sand-shaping.