RealtimeWAM: One-Step Asynchronous World Action Models

World action models are promising for robot control precisely because they bring a video model’s predictive representation to bear on action. That promise carries a latency bill, and most of it is paid twice: once when the action expert denoises in several steps, and again when the video and action experts take turns instead of sharing the machine.

Background

World action models (WAMs) incorporate visual representations from video generation backbones to guide action prediction. Recent efficient WAMs adopt Mixture-of-Transformers (MoT) architectures and compute the video representation once, for reuse by the action expert, which removes some redundant work. Two costs remain. First, intra-expert iteration: the action expert still performs multi-step denoising, so each step adds latency. Second, inter-expert waiting: the video expert and the action expert run sequentially, so one idles while the other works. Both costs are structural to the MoT design rather than incidental, and together they keep an otherwise efficient model out of real-time control.

Core Innovation

RealtimeWAM is a WAM variant built for one-step action generation and asynchronous inference, with a technique aimed at each bottleneck. To cut intra-expert iteration it introduces Teacher-Anchored Consistency Distillation (TACD), which targets a local-global error gap: low local consistency error alone does not guarantee accurate final actions, so TACD supplements local consistency with explicit supervision from the frozen teacher’s multi-step rollout endpoint. That extra anchor is what makes one-step generation accurate enough to be worth the speedup. To eliminate inter-expert waiting it introduces Cross-Expert Wavefront Pipelining (CEWP), which overlaps the video and action experts through block-wise sharing of the video KV cache, synchronizing only immediately before the corresponding action attention consumes it. The two techniques are complementary: TACD reduces how many steps the action side needs, while CEWP makes the remaining work overlap instead of serialize.

Results

• Across diverse benchmarks (LIBERO, LIBERO-Plus, RoboTwin) and model variants (Fast-WAM, Faster-WAM), RealtimeWAM maintains near-lossless performance, a drop of under 1%.
• It delivers a significant end-to-end speedup, on the order of 25x on an H100.
• Code and checkpoints are released through the LightX2V repository.

Limitations

The evaluation is reported as a relative performance drop (under 1%) rather than as absolute task-success figures, so the ceiling the method preserves is not visible from the abstract. The speedup is measured on an H100, and the abstract does not report latency on the embedded or edge hardware where real-time robot control usually has to run, so the on-robot implication is inferred rather than shown. The techniques are tailored to MoT-style WAMs with a decomposable video and action expert; whether the gains carry to other world-model architectures is not addressed.

Industry Implications

For robot teams, the decisive question about world action models is not whether they predict well but whether they run fast enough on the hardware a robot carries. RealtimeWAM’s contribution is squarely on that axis, and its most useful property is that the speedup is near-lossless and composes with existing MoT-style WAMs, so it can be adopted as an optimization rather than a redesign. Released code and checkpoints lower the barrier further. The caveat worth flagging for adopters is that the reported speedup is on datacenter-class GPUs; teams targeting edge deployment should re-measure on their own hardware before assuming the same ratio, and treat the near-lossless claim as a leaderboard-relative result to be re-checked on their own tasks.