XPACE: Joint World and Action Modeling from Heterogeneous Experience

· Editorial Team estimated
world-model world-action-model humanoid video-prediction heterogeneous-data policy-self-improvement

XPACE is a unified embodied world model that serves at once as a world action model, jointly predicting executable robot actions and future video, and as a world simulator that predicts the visual consequences of prescribed actions. A shared video backbone lets action-unlabeled video teach visual dynamics while action-labeled human and robot demonstrations jointly teach video and action prediction. A coarse-to-fine curriculum progressively shifts emphasis toward robot control while retaining human experience, so the policy acquires behaviors absent from robot demonstrations. The model then adapts its simulator to its own generated context, synthesizes deviation-recovery trajectories around expert demonstrations, and fine-tunes the policy on filtered recovery data. On XPENG's IRON humanoid, heterogeneous training improves robustness and transfers human-observed skills, and self-generated recovery data raises real-world task completion.

Paper · arXiv:2609.17372

A general-purpose robot has to do three things at once: draw on diverse experience, choose actions, and anticipate how those actions will reshape the world. Today these capabilities are usually split apart. A policy network produces actions; a world model or video predictor imagines the future; each is trained separately. The consequence is that the largest source of experience, action-unlabeled human video, cannot feed policy training directly, and even a model that can imagine the future has no clean way to turn those imagined futures into supervision for the policy.

Core Innovation

XPACE’s response is to make the two into one model. It is a unified embodied world model playing two roles at once:

  • World action model: jointly predicts executable robot actions and future video;
  • World simulator: predicts the visual consequences of a prescribed action.

The key design is that the policy and the simulator share a single video backbone. During training, action-unlabeled video teaches visual dynamics, while action-labeled human and robot demonstrations jointly teach video prediction and action prediction. The authors then add a coarse-to-fine training curriculum that progressively shifts emphasis from general visual dynamics toward robot control while retaining human experience, so the policy learns behaviors that robot demonstrations never covered.

The third piece is using the simulator to manufacture supervision. The model first adapts the simulator to its own generated context, then synthesizes deviation-recovery trajectories around expert demonstrations, and finally fine-tunes the policy on filtered recovery examples.

Results

  • Experiments on XPENG’s IRON humanoid robot: heterogeneous training improves robustness and enables transfer of human-observed skills to tasks absent from the robot demonstrations.
  • Recovery data produced by the model’s own simulator further improves real-world task completion.
  • Taken together, the results are presented as evidence that joint world and action modeling can connect learning from heterogeneous experience with simulation-driven policy self-improvement.

Limitations

The abstract reports no quantitative outcomes: no task counts, no success rates, no baselines. It does not detail the shared backbone architecture or how information is separated between the policy head and the simulator head. The error-accumulation risk of adapting a simulator to its own generated context before synthesizing recovery trajectories is not discussed. Model scale, training data volume, inference hardware, and whether artifacts are released are all omitted.

Industry Implications

The humanoid data bottleneck is not a shortage of video; it is a shortage of action-labeled video. XPACE treats human footage as a corpus for visual dynamics and a modest set of robot demonstrations as action supervision, which amounts to an architectural template for welding two very different data pools together. Using the model’s own simulator to generate recovery data addresses a second practical pain point: real robots rarely accumulate many failure and correction samples, yet that is exactly what policy robustness needs. For teams building a humanoid data flywheel, this structure offers a workable path to closing the loop across human video, robot demonstrations, and self-generated data.