Importance Sampling and PCA for Finding Failures in Commercial Autonomous Vehicles

Background

Safety validation for autonomous systems requires discovering rare failures that rarely appear in natural driving data. Existing rare-event discovery methods have been demonstrated almost exclusively on simple academic driving stacks, leaving open the question of whether they generalize to the more robust planners used in commercial systems. This gap means the industry cannot assess the practical effectiveness of these approaches.

Core Innovation

This paper applies two rare-event discovery algorithms — Adaptive Stress Testing (AST), which uses reinforcement learning to search for noise trajectories most likely to cause collisions, and Diffusion-based Failure Sampling (DiFS), which trains a denoising diffusion model to sample diverse failures — to a real commercial autonomous trucking stack. Both discover simulated collisions during merge and cut-in maneuvers that traditional Monte Carlo simulation cannot find.

The key methodological contribution is a PCA-based statistical analysis pipeline: principal component analysis classifies discovered failures into common modes, identifies influential timesteps, and inverts the PCA transform to recover generalized noise trajectories that reproduce failures in identical and similar scenarios. This creates a complete path from failure discovery to systematic diagnosis of perception-level flaws.

Results

On the commercial trucking stack, both AST and DiFS successfully discover diverse collision types during merge and cut-in maneuvers where Monte Carlo simulation yields zero collisions. PCA clustering effectively distinguishes failure modes, and inverted trajectories reliably reproduce failures across similar scenarios. The results confirm that these methods transfer from academic stacks to real commercial systems.

Limitations

The experimental scenarios are limited to highway merge and cut-in maneuvers. PCA analysis effectiveness depends on the diversity and scale of discovered failures. The full attribution chain from inverted noise trajectories to specific perception-level defects needs further automation.

Industry Impact

This work directly bridges rare-event failure discovery from academic simulation to commercial systems, providing a valuable addition to the AV safety validation toolbox. The PCA-based systematic diagnosis pathway offers an actionable approach for empirical safety论证 during ODD expansion.