Meaningful Change Detection

Speaker: Eric Martinson

Repeated visual observations of an environment are common in big data: people capturing temporally separate video streams with phones; facility security combining fixed cameras with human patrols; robots cleaning or monitoring a home. The challenge, however, is effectively processing these large highly repetitive data to extract useful results. Event-based methods like object detection struggle with a lack of application specific training data, while anomaly-based methods have high false positive rates requiring significant human review. Indoor spaces further complicate the matter as they are often co-occupied by people, changing constantly, and have highly individual detection requirements. What is needed are new ways for incorporating context into the search, discarding that which a human observer would otherwise ignore. To address this challenge, we have developed a novel system for Meaningful Change Detection, integrating two recent advances in machine learning: Neural Radiance Fields (NeRF’s) and Contrastive Language-Image Pre-Training (CLIP). Combining these approaches allows us to generate before and after images from the same viewpoint with a NeRF model, then apply semantically meaningful queries to search for changes useful to the application. This talk will present early results from the first prototype system and discuss future directions for investigation.