Is It Safe to Cross? Interpretable Risk Assessment with GPT-4V for Safety-Aware Street Crossing

Abstract

Safely navigating street intersections is a complex challenge for blind and low-vision individuals, as it requires a nuanced understanding of the surrounding context - a task heavily reliant on visual cues. Traditional methods for assisting in this decision-making process often fall short, lacking the ability to provide a comprehensive scene analysis and safety level. This paper introduces an innovative approach that leverages vision-language models (VLMs) to interpret complex street crossing scenes, offering a potential advancement over conventional traffic signal recognition techniques. By generating a safety score and scene description in natural language, our method supports safe decision-making for blind and low-vision individuals. We collected crosswalk intersection data that contains multiview egocentric images captured by a quadruped robot and annotated the images with corresponding safety scores based on our predefined safety score categorization. Grounded on the visual knowledge, extracted from images and text prompts, we evaluate a VLM for safety score prediction and scene description. Our findings highlight the reasoning and safety score prediction capabilities of the VLM, activated by various prompts, as a pathway to developing a trustworthy system, crucial for applications requiring reliable decision-making support.