Driving on Memory

End-to-end autonomous driving models plan future trajectories directly from raw sensor input. Earlier benchmarks measured deviation from a human trajectory; modern benchmarks such as NAVSIM and Bench2Drive use richer simulation-based metrics intended to capture safe and compliant driving. The implicit assumption is that a high score reflects genuine scene understanding — that the model sees what is in front of it and acts accordingly. This paper asks how much of that score actually comes from reacting to the dynamic part of the scene.

Core Innovation

The experimental design is elegant in its simplicity: remove the camera, substitute memory. The authors replace a model’s camera input with memories retrieved from prior drives at the same location. These memories provide persistent scene information — road layout, location-conditioned regularities — but crucially, no information about the current traffic state.

If a model still scores highly, its benchmark performance was never about reacting to the traffic in front of it. The paper also tests whether the effect holds across benchmarks, comparing NAVSIM against Bench2Drive and RealEngine.

Results

• NAVSIM: memory is nearly sufficient — performance reaches or even exceeds leading end-to-end methods without observing the evaluated scene.
• Benchmark dependence: driving from memory causes substantially larger performance drops on Bench2Drive and RealEngine, indicating those benchmarks do require reacting to the current scene.
• Interpretation: a high NAVSIM score should be treated with caution; it does not necessarily require a planner to react to current traffic.
• Openness: code is provided at github.com/boschresearch/MemoryDrivoR.

Limitations

The paper is an evaluation study rather than a new driving method, so it does not improve driving performance itself. Memory retrieval assumes prior drives at the same location exist — a condition that does not hold for novel or rarely visited routes. The precise mechanism by which NAVSIM rewards location-conditioned regularities is identified behaviorally but not fully decomposed into metric-level contributions.

Industry Implications

For teams benchmarking end-to-end driving models — from OEMs to autonomy startups — this is a cautionary and immediately actionable result. If NAVSIM scores can be inflated by location memory, model rankings on that benchmark may not reflect reactive driving skill; complementary evaluation on Bench2Drive-style benchmarks or scenario-based testing becomes essential. For the broader evaluation ecosystem, it argues for reporting multiple benchmarks with different sensitivity profiles, and for treating leaderboard scores as evidence about scene priors as much as about driving competence.