CLAP: Cross-Embodiment Video World Models are Zero-Shot Physical Simulators

State-of-the-art action-conditioned video models are almost always tied to a single robot embodiment. That constraint is self-limiting: the videos such models can learn from are a tiny fraction of the heterogeneous video that actually exists, even though the physical rules governing how objects move, collide, and deform are the same whether the actor is a human hand or a robot arm. CLAP is built on the observation that universal physical laws govern spatiotemporal dynamics regardless of the actor — and on the engineering required to exploit that fact.

Core Innovation

CLAP addresses the fundamental obstacle to cross-embodiment learning: action representations vary sharply across robot platforms and are usually absent from human video altogether. Its approach has two pillars:

• Unified action spaces — the framework reconciles disparate action spaces using three complementary representations: end-effector poses, language instructions, and latent actions, each covering the blind spots of the others.
• Curriculum-based cross-embodiment learning — the model first learns foundational physical priors from large amounts of unlabeled video using latent actions, then grounds these priors in end-effector action spaces, enabling zero-shot deployment to real-world tasks.

The result is a single family of video world models that can be trained on data spanning humans, bimanual platforms, mobile manipulators, and humanoids.

Results

• DROID: CLAP approaches or surpasses state-of-the-art single-embodiment video models in this challenging environment.
• Few-shot adaptation: the cross-embodiment advantages compound with few-shot adaptation, establishing a new paradigm for training single-embodiment video world models.
• Breadth: the released suite covers end-effector, language, and latent action-conditioning across DROID, Bridge, bimanual YAM robots, and G1 humanoids — the most comprehensive action-conditioned video world model suite reported to date.
• Openness: all code and models are open-sourced, with a project website at omni-clap.github.io.

Limitations

The abstract characterizes DROID results as “approaches or surpasses,” so the margin over single-embodiment models is not uniform across every benchmark. Training at internet scale with a curriculum across heterogeneous action spaces implies substantial data and compute requirements that are not quantified. The zero-shot story is strongest where actions can be cleanly grounded in end-effector or language space; performance under extreme visual novelty or contact-rich dynamics is not detailed.

Industry Implications

Cross-embodiment world models change the economics of robot learning. A single video world model trained on human plus robot data can serve as a physical simulator for planning, policy pretraining, and synthetic data generation across humanoids, arms, and mobile manipulators — reducing the per-embodiment data collection burden that currently dominates foundation-model development. For companies building general-purpose robots, this points toward treating unlabeled internet video as a first-class training resource rather than an afterthought.