SpatialDex: 3D-Grounded Bimanual Dexterity from RGB-Only Human Video

Half of the dexterous manipulation data bottleneck is artificial. Human video is abundant, and hand-pose annotation is increasingly reliable, yet policies pretrained on it often disappoint on real robots. The problem is not the motion — it is the space. A hand in a video moves through a specific three-dimensional scene: how tall the table is, whether the cup sits left or right of the hand, how much clearance the wrist has from an obstacle. Most pipelines extract the hand trajectory and throw the geometry away. The policy learns how the hand should move but not where in space that movement happens.

Core Innovation

SpatialDex learns human behavior together with its spatial context. For each video it:

1. reconstructs the scene geometry and estimates hand motion;
2. aligns the two at a shared video-level scale using the hand-size prior that comes with 3D hand estimates — this addresses the scale ambiguity of monocular reconstruction and is what lets human-video geometry share a coordinate frame with robot data;
3. keeps the reconstructed geometry as both a policy input and a prediction target, not merely a visualization aid.

Architecturally, a pretrained vision-language model serves as the backbone, and gated residual fusion grounds its token grid in the reconstructed geometry. The training objective goes beyond action prediction: the model predicts future scene geometry while emitting wrist and finger actions, and finger motion is generated conditioned on the predicted trajectories of both wrists. That decomposition — decide where the wrists go, then how the fingers articulate — matches the physical causal order of dexterous manipulation.

Pretraining uses 1M human-video clips, followed by fine-tuning on only a small number of robot demonstrations, which is the method’s central claim about data economics.

Results

• +16.5 points over the prior state of the art on nine simulated dexterous tasks across two embodiments;
• +38.3 points on six real-world bimanual dexterous tasks;
• In a matched comparison, pretraining with reconstructed geometry beats pretraining on hand motion alone by 14.2 points in success rate.

That last ablation is the most informative: it attributes the gain to geometry itself rather than to pretraining volume or model capacity.

Limitations

• Only relative scores are reported; absolute per-task success rates are absent, as is any description of real-world task difficulty (object counts, contact richness, occlusion).
• The method depends on monocular 3D hand estimation and scene reconstruction; how reconstruction error propagates to the policy under heavy occlusion, fast motion or reflective materials is not evaluated.
• The size of the “small number” of robot demonstrations is unspecified, and it determines practical usability — if hundreds of demonstrations are needed, collection cost eats the pretraining advantage.
• Evaluation is confined to tabletop bimanual dexterity; mobile platforms, large workspaces and long-horizon multi-object tasks are untested.

Industry Implications

The real contribution is an upgrade in status for human video: from motion data to spatial supervision. For any robotics company accumulating data, assets that previously yielded only hand trajectories can now also supply scene geometry and hand-object spatial relations — one data source, two uses.

The design also reflects an emerging consensus: a VLM backbone supplies semantic priors, geometry supplies spatial precision, and the two are joined by lightweight fusion (here, gated residuals) rather than by forcing point clouds into the token stream. That is friendlier to compute budgets and makes existing vision-language models reusable.

The question worth watching is whether the 14.2-point geometry gain grows with data scale. If geometric supervision does not saturate as video volume increases, “reconstruct and predict future geometry” becomes a standard component of human-video pretraining. If it saturates quickly, the bottleneck lies in action coverage rather than spatial representation.