DreamWAM: Beyond RGB Future Prediction for World Action Models

Background

World action models (WAMs) learn representations by predicting how the observed world will evolve under an agent’s actions, and those representations feed action generation. Most existing WAMs predict the future in RGB space, where task-relevant state transitions are entangled with nuisance variation — texture, illumination, background, and viewpoint. The result is that a model can be good at predicting pixels while being poorly grounded in the state changes that actually matter for control. The paper’s core argument is that robust world-action learning requires predicting the future in a form that matters for action, not merely in RGB.

Core Innovation

DreamWAM reformulates future prediction as structured world modeling beyond RGB: future states are represented through complementary views of appearance, motion, geometry, and semantics. During training, the model jointly denoises RGB and motion latents and adds lightweight gated residual branches for geometry and semantics. Shared attention between the video branch (VideoDiT) and the action branch (ActionDiT) lets the action policy learn from these future-state predictions. A deployment-friendly property is that all beyond-RGB supervision branches are disabled at inference — the model runs RGB-only, so the added supervision improves the learned representation without adding runtime cost.

Results

Across no-rollout and joint video-action inference, DreamWAM consistently improves over matched RGB-only baselines on LIBERO, from 97.30% to 98.40% and from 98.00% to 98.90% respectively. The gains widen substantially under unseen LIBERO-Plus perturbations: from 51.36% to 63.44% and from 69.16% to 75.47%. The robustness transfers to real-world manipulation, where DreamWAM attains a 74.4% average success rate across unseen changes in lighting, background, and object layout, versus 55.6% for the Fast-WAM-Joint baseline. Code and models are publicly released, which is a strong reproducibility signal.

Limitations

The reported gains, while consistent, are concentrated in robustness-oriented evaluations and the paper’s own benchmark configurations; independent replication on standard manipulation suites would strengthen the claims. The real-world evaluation is summarized at the aggregate level, so per-condition variance (e.g., which perturbation types are hardest) is not visible from the abstract. The architecture adds training-time complexity with multiple supervision branches, and the practical benefit of each branch beyond motion is not detailed here.

Industry Implications

Robustness to visual perturbations is the difference between a manipulation policy that works in the lab and one that survives a factory floor or a home. DreamWAM’s result — that structured multi-modal future prediction during training yields substantially better out-of-distribution performance at zero inference-time cost — is directly actionable for teams training manipulation policies. The public code and model release lowers the barrier for adoption, and the RGB-only deployment profile means the approach does not force hardware or sensor changes on existing systems. For VLA and world-model practitioners, this is a concrete argument for supervising beyond-RGB state during training.