WorldLine: A Video World Model That Predicts Manipulation Before the Robot Moves

A world model is only worth building if its predictions change what you do next. The test for a visual simulator is not whether the imagined frames look real, but whether acting on them avoids a failed trial and picks a better policy.

Background

Real-world robot learning is limited as much by evaluation as by data collection: every candidate behaviour is expensive to run on hardware. A video generator that could simulate the consequences of an action before it is executed would let a policy iterate in imagination, but today’s models disappoint in two ways. They tend to prize visual plausibility over faithful action-following and coherent robot-object interaction, so the imagined future looks good but does not match what the action would actually do. And the models that do condition on actions need scarce, embodiment-specific data that is hard to share across robots with incompatible control spaces. The result is a simulator that is either pretty but unfaithful, or faithful but unshareable.

Core Innovation

WorldLine resolves this by decoupling the two halves of the problem: transferable dynamics learning on one side, heterogeneous action grounding on the other. It learns manipulation dynamics from more than 10,000 hours of action-free robot video - footage that is abundant precisely because it carries no action labels - and then grounds those dynamics using over 2,000 hours of action trajectories spanning more than ten embodiments. The bridge between the halves is an image-space action representation that serves as a shared control interface, a common language into which different robots’ control signals can be mapped. To make prediction sensitive to the moments that matter, WorldLine adds multi-view and failure-enriched training with relational regularization, so interaction-heavy, failure-prone segments are modelled more carefully. A robot-focused few-step distillation step finally makes causal rollout efficient enough to be used routinely rather than as a demo.

Results

• On failed trajectories, WorldLine improves robot-mask IoU by 0.1626 over the strongest baseline, a direct measure of how well it renders the robot where it actually goes.
• It predicts trajectory success with 74% mean accuracy across RoboTwin and AgiBot, one percentage point above the strongest baseline.
• Without any RoboTwin training or adaptation, its rollouts improve downstream task success by up to 21.4 percentage points over executing the policy directly.
• Across held-out and out-of-domain settings it maintains strong visual quality and robot-motion agreement, and few-step distillation preserves the action-critical motion needed for causal rollout.

Limitations

Success prediction tops out around 74%, so as a filter for whether a trajectory will work it is useful but not decisive: roughly a quarter of predictions are wrong. The gains are relative to specific baselines on specified benchmarks, and the headline 21.4-point improvement is measured without RoboTwin training, which is a strong sign of transfer but also means the number is tied to that particular experimental setup. The system depends on a very large action-free video corpus and thousands of hours of trajectories; teams without comparable data cannot simply adopt the recipe. And a visual simulator models appearance and motion, not contact forces or hidden physics, so tasks dominated by force-sensitive or deformable interactions sit outside what it can credibly predict.

Industry Implications

If a simulator can score candidate actions before they touch hardware, it changes the economics of robot development: policy iteration shifts from physical trials, which are slow and consume robot hours, to imagined trials, which are cheap and parallel. The decoupled design is the practical part - unbundling a large pool of unlabelled video from the smaller set of action-labelled trajectories means the expensive labelled data is amortised across many embodiments, rather than recollected per robot. For teams shipping policies, the immediate use is regression testing and safety screening: run the candidate in the simulator first, catch the failures that look fine in aggregate metrics, and only spend real robot time on plausible candidates. The residual error in success prediction argues for treating it as a triage tool rather than an oracle.