Spatially Aware World Action Model via Geometric Latent Diffusion

· Editorial Team estimated
world-models robot-learning video-diffusion depth-perception manipulation

SA-WAM repurposes a pretrained video diffusion model into a spatially aware World Action Model that predicts actions, RGB, and depth within a single diffusion backbone. A nonlinear depth encoding maps unbounded depth into the bounded input domain of the frozen VAE tokenizer, adding geometric awareness without 3D-specific fine-tuning. SA-WAM achieves state-of-the-art results on RoboCasa and LIBERO-Plus while improving future-state predictions, and outperforms strong baselines on a real UR5 arm with large gains in randomized environments.

Paper · arXiv:2609.02531

World Action Models (WAMs) built on large pretrained video diffusion models predict future observations and actions together, inheriting rich visual and physical priors from internet-scale video. They have become a promising paradigm for robot policy learning. Yet nearly all of them operate purely on RGB: they never see the 3D structure of the scene, even though manipulation is fundamentally a geometric problem. SA-WAM asks whether a world action model can gain spatial awareness without throwing away the pretrained priors that make it work.

Core Innovation

  • Joint action, RGB, and depth prediction — a pretrained video diffusion model is repurposed so a single backbone predicts actions, future RGB frames, and future depth in one pass.
  • Frozen-tokenizer geometric encoding — a nonlinear encoding maps the unbounded depth signal into the bounded input domain expected by the frozen VAE tokenizer, letting depth enter the model with no 3D-specific fine-tuning and no loss of pretrained priors.
  • Prediction-quality analysis — the paper studies the correlation between world-model prediction quality and downstream rollout success, offering concrete pointers on where WAM gains actually come from.

Results

  • State-of-the-art results on the RoboCasa and LIBERO-Plus benchmarks, while simultaneously improving future-state prediction quality.
  • Outperforms strong baselines in real-world evaluation with a UR5 robotic arm, with the largest gains in randomized environments.
  • Ablations link better world-model prediction to better task success, clarifying when geometric awareness helps most.

Limitations

The results reported in the abstract concentrate on tabletop and kitchen-style manipulation with a single-arm real-robot setup, so generalization to mobile or humanoid platforms remains open. Deployment assumes depth is available at inference time, either from an RGB-D sensor or an upstream estimator, and the bounded-domain encoding may compress very close or very distant geometry. No failure-mode analysis or deployment-scale study is described in the abstract.

Industry Implications

3D-aware world models speak directly to the economics of robot learning: if a pretrained video foundation model can be given geometric grounding with no 3D fine-tuning, teams building generalist manipulation policies can spend far less on task-specific data collection. The real-robot gains on a UR5 — especially in randomized, less controlled setups — point toward industrial pick-and-place and service-robot manipulation where scene geometry constantly varies. For platform vendors, the depth-at-the-tokenizer trick is an attractive blueprint: spatial awareness added at the input encoding level, preserving the full value of existing pretrained backbones.