RoboSPA: Can VLA Models Go Beyond Simple Scenes and Short-Horizon Tasks?

· Editorial Team estimated
VLA benchmark embodied-AI spatial-reasoning long-horizon

RoboSPA (Robot Spatial-Procedural Assessment) is a large-scale robotic manipulation dataset and diagnostic benchmark for embodied reasoning in VLA models. It focuses on two core dimensions — fine-grained spatial reasoning and long-horizon procedural planning — covering 10 task categories and 56 base tasks, each instantiated across five difficulty levels to yield 280 variants with increasing spatial ambiguity and procedural complexity. The dataset contains 527K trajectories across multiple embodiments and diverse scenes. Beyond binary success rate, RoboSPA introduces diagnostic metrics for finer-grained evaluation. Experiments on representative VLA models show current systems still struggle with complex spatial relations, precise low-level execution, and memory-intensive planning, establishing RoboSPA as a challenging diagnostic benchmark for more capable, reliable, and generalizable embodied agents. Data and code are public.

Paper · arXiv:2609.05324

Vision-Language-Action (VLA) models have made striking progress in language-conditioned manipulation, but existing datasets and benchmarks mostly ask whether a task was completed under predefined settings. They offer little insight into how a model’s reasoning degrades as spatial relations tighten and procedural chains lengthen. Near-perfect scores on saturated benchmarks like LIBERO can mask exactly those weaknesses. RoboSPA is built to diagnose where such high-scoring models actually fail.

Core Innovation

RoboSPA shifts evaluation from task completion to two core capabilities: fine-grained spatial reasoning and long-horizon procedural planning. It comprises 10 task categories and 56 base tasks, each instantiated across five difficulty levels — 280 variants with deliberately increasing spatial ambiguity and procedural complexity. The dataset spans 527K trajectories across multiple embodiments and diverse scenes. Instead of reporting only binary success, RoboSPA introduces diagnostic metrics designed to localize which reasoning stage degrades: understanding of spatial relations, precise low-level execution, or memory-intensive planning.

Results

  • Systematic evaluation of representative VLA models shows they still struggle with complex spatial relations, precise low-level execution, and memory-intensive planning.
  • Diagnostic metrics separate failure modes along different capability dimensions rather than collapsing everything into a single success number.
  • Data and code are public at github.com/fanzhenxuan/RoboSPA.

Limitations

The abstract does not name which VLA models were evaluated or give per-difficulty score distributions, so the quantitative gap landscape needs the full paper. The provenance of the 527K trajectories (teleoperation, demonstration, or simulation) and the embodiment mix are not specified in the abstract, and the five-level difficulty calibration carries some subjectivity. As a pure benchmark contribution it offers no improvement method; its value depends on the community iterating on the diagnosed weaknesses.

Industry Implications

For companies betting on embodied AI, benchmarks are roadmaps. RoboSPA converts the industry-wide anxiety about inflated manipulation success rates into localized, actionable diagnoses of reasoning failures, giving data-collection and model-selection teams a finer-grained yardstick. Its “graded difficulty plus diagnostic metrics” evaluation paradigm transfers directly to in-house data flywheels and model acceptance testing, surfacing genuine spatial-reasoning and long-horizon memory gaps before deployment.