Pelican-Sim 1.0: A General World Model Simulator for Embodied Intelligence

· Editorial Team estimated
world-model embodied-ai unified-action-space mixture-of-experts diffusion-distillation trajectory-generation

Pelican-Sim 1.0 is a general world model simulator for embodied intelligence: it predicts future observations from visual context and robot actions to support downstream learning and decision making. Four design choices carry the system — a 28-dimensional action value space covering most mainstream embodiments so that one model stays valid across heterogeneous devices, action-visual injection that bridges actions and pixels with URDF- and camera-rendered action videos, sparse mixture-of-experts layers that absorb the action modality while reducing inter-modality conflict, and a four-step autoregressive simulator produced by causal adaptation and few-step distillation. Trained on roughly one million real-world and simulated trajectories, it improves video quality and action controllability substantially; in four downstream RoboTwin applications, adding 500 generated trajectories per task to 50 demonstrations raises policy success from 70% to 93%.

Paper · arXiv:2609.12036

Embodied AI is bottlenecked by the cost of real data: collecting demonstrations at scale is limited by hardware, and evaluating policies means repeatedly returning to a real environment. World models are the promised remedy — if a model can accurately predict future observations from visual context and actions, it can take on both data generation and policy evaluation.

Core Innovation

Pelican-Sim 1.0, presented as a technical report, is built around four features:

  1. Unified action representation: a 28-dimensional action value space spanning most mainstream embodiments, keeping one model valid across heterogeneous devices instead of modelling each robot separately.
  2. Action-visual injection: URDF- and camera-rendered action videos bridge actions and pixels, giving markedly better controllability across embodiments, scenes, and tasks (PSNR +0.904 over alternative fusion baselines).
  3. Sparse mixture-of-experts: sparse MoE layers add capacity for heterogeneous dynamics and absorb the action modality, reducing inter-modality conflict (FVD −6.530 versus the dense backbone).
  4. Efficient rollout generation: causal adaptation and few-step distillation yield a four-step autoregressive simulator, a 5.67x speedup over the 35-step model.

Results

  • Trained on approximately one million real-world and simulated trajectories.
  • Large gains in video quality and action controllability: PSNR improves over the strongest evaluated baselines by +4.636 on AgiBotWorld Beta, +2.080 on RoboMIND, and +10.343 on RoboTwin, with the adapted EWMBench DYN score up 0.426 on RoboTwin.
  • Four downstream RoboTwin applications succeed: 500 generated trajectories added to 50 demonstrations per task raise policy success from 70% to 93%; policy evaluation reaches a 0.994 Pearson correlation across five checkpoints; relative success gains reach 47.7% for action selection and 20.3% for policy improvement.
  • Qualitative generalization is shown across trajectory, scene, object, embodiment, and viewpoint shifts.

Limitations

This is a technical report whose abstract contains no real-robot deployment or closed-loop hardware experiment; the evidence is confined to simulated datasets and benchmarks, and the +10.343 PSNR gap on RoboTwin suggests strongly domain-dependent payoffs, leaving the generalization boundary unclear. The 28-dimensional action space is claimed to cover “most mainstream embodiments” without stating which embodiments or task types fall outside it, and cross-embodiment quality is not quantified. All four downstream uses are validated inside the RoboTwin ecosystem, so transfer to other simulators or real data pipelines remains unproven; the abstract also does not say whether code, weights, or data are released.

Industry Implications

The most realistic industrial entry point for world models is replacing expensive data collection and real-robot evaluation. Substituting part of policy evaluation at a 0.994 correlation directly shortens iteration cycles and cuts experiment cost, while “500 generated trajectories lift success from 70% to 93%” shows the model is already usable for data augmentation — especially attractive for long-tail tasks where data is scarce. The 5.67x speedup from a four-step autoregressive rollout is what determines whether such a model can sit inside a training loop rather than only in offline batch jobs. That one model stays valid across embodiments also means robot vendors need not train a separate world model per machine.