AI Insight
ExceptionDrive is a new benchmark for evaluating autonomous driving planners under rare, safety-critical scenarios by inserting hazards into real driving scenes from the nuScenes dataset. The benchmark includes 21 tasks across six safety categories and uses reference-free evaluation metrics to assess how well planners avoid hazards and maintain appropriate clearance distances. Testing seven existing planners revealed frequent failures to avoid hazard regions or maintain sufficient safety margins, while a proposed Reminder Agent system improved hazard detection and response strategy selection in zero-shot experiments.
Why it matters
Current autonomous driving systems are primarily evaluated on routine scenarios, which may not reveal dangerous failures in rare but critical situations. This benchmark provides a systematic way to test and improve planner behavior in edge cases that are most likely to cause accidents, potentially leading to safer autonomous vehicle deployment.
Understand the Science
⚠️ Preprint – Noch nicht peer-reviewed
Dieser Artikel wurde noch nicht von unabhängigen Experten begutachtet. Die Ergebnisse sind vorläufig und sollten mit Vorsicht interpretiert werden.
Abstract: Average performance on routine driving benchmarks does not establish planner reliability under rare, safety-critical hazards. We proposed ExceptionDrive, a counterfactual planning benchmark that uses VLM-assisted screening, localized multi-view editing, and quality auditing to insert hazards into real nuScenes scenes while preserving their context. Its 21 tasks span six safety families and define hazard or conflict regions, local safety constraints, and acceptable responses. Because hazard insertion can invalidate the recorded human trajectory, our reference-free protocol evaluates edited predictions using Unsafe Rate (UR), Hazard Clearance Compliance (HCC), Hazard Proximity Response (HPR), and Counterfactual Trajectory Shift (CTS), which measure core-region intrusion, clearance compliance, clearance relative to a prescribed margin, and counterfactual trajectory change. Seven representative planners frequently intrude into hazard regions or provide insufficient clearance. We also develop a Reminder Agent that, without sample-specific task labels, converts visual evidence and the shared taxonomy into structured records of hazard presence, type, and a recommended high-level strategy. The agent neither predicts trajectories nor controls the vehicle; its records guide a VLM-based decision agent. In zero-shot experiments, the reminders improve strategy accuracy and reduce under-warning.
Source: ExceptionDrive: A Planning-Oriented Counterfactual Corner-Case Benchmark for Autonomous Driving