Meaningful Change Detection in Indoor Environments Using CLIP Models and NeRF-Based Image Synthesis

Abstract

Security operations are all about detecting change. Looking for out of place or suspicious things or people is the job, so the first step is to learn what is normal and then recognize what is not. Change detection in robotics, however, has focused on the big picture - extracting mask images of new buildings or construction to support autonomous cars, or correcting semantic maps. If we want robots to help patrol a facility, a different type of change detection is required that can be quickly adapted for working with humans to address new security concerns. To this end, we propose a highly dynamic change detection system based on Contrastive Language-Image Pre-Training (CLIP) and Neural Radiance Fields (NeRF). NeRF is used to generate images from the viewpoint that are high quality indoor reconstructions, while CLIP-based segmentation allows a security guard to search for a variety of potential threats using natural language queries. The resulting robotic system is demonstrated to be effective in an office environment with no additional manual annotation.