One World State, Many Robots: A Shared-Space Action Model for Multi-Robot Teams

Multi-robot manipulation is usually attempted by extending a single-robot policy in the simplest way available: concatenate the camera feeds from every robot, hand the stack to a bigger model, and predict one action sequence per robot. It is easy to implement and it tends to plateau. The reason is that stacking views does not create a shared spatial frame. Two cameras watching the same table produce two sets of pixels with no mechanism that says this patch of image A is the same physical location as that patch of image B. Without that correspondence, more views mostly add parameters and latency.

Meanwhile the field has split into two camps. Vision-language-action models inherit semantic knowledge from pretrained VLMs but treat the future only implicitly. World-action models predict what happens next, gaining physical grounding, but usually at the cost of a video generator in the loop. Both families, as the authors point out, are overwhelmingly single-robot.

Core innovation

World Dreamer v5-lite is organized around a single shared world state, with robot-specific action prediction layered on top.

One state in workspace coordinates. Calibrated features from a pretrained video encoder are lifted into workspace coordinates to populate the shared state. This is the key design decision: because the state lives in a common spatial frame rather than in per-camera image space, adding a robot adds observations to the same representation instead of adding a parallel one.

A fixed spatial budget. The state’s spatial size stays constant as views are added, so a fourth or fifth camera does not blow up the token count. Cost scales with robots, not with the product of robots and views.

One query per robot, exchanging pose-aware information. A vision-language model supplies scene context and issues a query token per robot. The queries read the shared state and exchange pose-aware information with each other, while their action streams remain separate. Each robot therefore plans its own motion but conditions on what the team collectively knows.

Masked-view cooperative training. Selected camera views are withheld from both visual branches during training while all action targets are retained. This forces the policy to accomplish the task using observations contributed by other robots, which is precisely the behavior a team needs when one robot’s view is occluded by its own arm or by a teammate.

Scale of pretraining. Over 10,000 hours of trajectories from 20 simulated robot types, RoboFactory, and real robots — the breadth that makes a shared cross-embodiment state plausible rather than a two-robot special case.

Results

• RoboFactory: 74.0% mean success, 23.7 percentage points above the strongest evaluated baseline.
• Four real-robot tasks: 78.8% mean success, a 13.8 point margin.
• RoboCasa365: 58.08% overall, which places it on a substantially harder and broader benchmark than the two headline numbers.

The margin structure is the interesting part. A 23.7 point gain in simulation that shrinks to 13.8 points on hardware is the normal and honest pattern — simulation flatters every method — but both margins are large enough that the effect is unlikely to be a tuning artifact. Reporting RoboCasa365 at 58.08% rather than omitting it is also a signal of confidence: it is the number that makes clear the system is strong, not solved.

Limitations

The abstract gives no per-task breakdown of the four real-robot tasks, so it is impossible to tell whether the 78.8% average is carried by two easy tasks and two near-failures. No ablations are reported for the components that matter most: how much of the gain comes from the shared workspace-coordinate state versus the masked-view training versus simply having more parameters than the baselines. Without that decomposition, the central claim — that spatial correspondence rather than extra views drives the improvement — is plausible but not isolated. The method assumes calibrated cameras and known relative poses, which is a real constraint in a team of mobile robots that must re-localize as they move; nothing is said about recalibration or pose drift. Nor is anything reported about communication cost, latency, or behavior when a teammate’s view is lost entirely rather than merely masked during training.

Industry implications

Warehouses and assembly cells are moving toward multiple manipulators sharing a workspace, and the current integration pattern — several independently trained policies plus a hand-written scheduler — is brittle precisely where coordination matters. A shared world state in workspace coordinates offers a cleaner interface: each robot keeps its own action head and its own embodiment-specific controller, while the spatial understanding is common. Adding a robot becomes a calibration and enrollment step rather than a retraining project.

The masked-view training detail has an underappreciated operational payoff. In real cells, cameras get occluded constantly — by an arm, by a carried part, by a coworker. A policy trained to complete tasks from teammates’ observations degrades gracefully in exactly those moments, which is the difference between a demo and a shift-long deployment.

Two caveats temper this. The 10,000 hour pretraining corpus is out of reach for most integrators, so the practical value depends on whether this generalizes as a foundation model that others fine-tune — which the RoboCasa365 number supports but does not prove. And the reliance on calibrated extrinsics is a maintenance burden in any facility where robots or cameras get moved. A version that tolerates calibration drift would be considerably more deployable than a higher success rate on a fixed benchmark.