ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and\n Tactile Representations

Abstract

Multisensory object-centric perception, reasoning, and interaction have been\na key research topic in recent years. However, the progress in these directions\nis limited by the small set of objects available -- synthetic objects are not\nrealistic enough and are mostly centered around geometry, while real object\ndatasets such as YCB are often practically challenging and unstable to acquire\ndue to international shipping, inventory, and financial cost. We present\nObjectFolder, a dataset of 100 virtualized objects that addresses both\nchallenges with two key innovations. First, ObjectFolder encodes the visual,\nauditory, and tactile sensory data for all objects, enabling a number of\nmultisensory object recognition tasks, beyond existing datasets that focus\npurely on object geometry. Second, ObjectFolder employs a uniform,\nobject-centric, and implicit representation for each object's visual textures,\nacoustic simulations, and tactile readings, making the dataset flexible to use\nand easy to share. We demonstrate the usefulness of our dataset as a testbed\nfor multisensory perception and control by evaluating it on a variety of\nbenchmark tasks, including instance recognition, cross-sensory retrieval, 3D\nreconstruction, and robotic grasping.\n