ModAR: Modality-Autoregressive World-Action Models

· Editorial Team estimated
world-action-model multimodal-prediction point-tracks dino-features depth-maps training-efficiency

World-action models (WAMs) jointly model future observations and actions, and they almost always represent the predicted future as RGB images; yet depth maps, pretrained visual features and point tracks capture geometric, semantic and motion information far more efficiently. How best to combine these modalities inside a WAM has remained open. ModAR is the first WAM to autoregressively denoise multiple future modalities before predicting actions, so every prediction conditions on the modalities generated so far. Trained from scratch, it systematically studies training-data mixtures, which modalities to predict, and WAM formulations. The findings are that predicting point tracks, DINO features and depth maps helps, while additionally predicting future RGB gives no consistent benefit. ModAR's sequential generation attains the highest average success rate at every evaluated data scale, and when a video-model-initialized WAM, Flex-pi, is fine-tuned on the same data, ModAR reaches a slightly higher observed average success rate (75% vs 72%) using roughly 20x fewer training FLOPs and no pretraining. On three real-world bimanual tasks it beats baselines and improves further with human videos.

Paper · arXiv:2609.17524

World-action models (WAMs) have converged on a standard recipe: jointly predict future observations and actions, and represent the future almost exclusively as RGB images. That default is rarely questioned, even though RGB is not necessarily the most efficient carrier of information. Depth maps encode geometry directly, pretrained visual features such as DINO carry semantics, and point tracks describe motion; all three can pack more relevant signal per unit of prediction than pixel reconstruction. The real question is therefore not which modality to use, but how to combine and order these modalities inside a single WAM.

Core Innovation

ModAR answers with sequential generation. It is the first WAM to autoregressively denoise several future modalities before predicting actions, with each step conditioned on the modalities already generated. The model can thus unfold the future one modality at a time and only then commit to an action.

To ground its conclusions in controlled evidence, the authors train from scratch and sweep three variables:

  1. training-data mixture;
  2. which modalities are predicted (combinations of point tracks, DINO features, depth maps and RGB);
  3. the WAM formulation (sequential generation versus existing alternatives).

Results

  • Modality choice: predicting point tracks, DINO features and depth maps all help; adding future RGB on top brings no consistent benefit, a direct counterexample to the assumption that a WAM must reconstruct pixels.
  • Model form: ModAR’s sequential generation achieves the highest average success rate at every evaluated data scale.
  • Training efficiency: against a video-model-initialized WAM (Flex-pi) fine-tuned on the same data, ModAR reaches a slightly higher observed average success rate of 75% (versus 72%) using roughly 20x fewer training FLOPs and no pretraining.
  • Real robots: on three real-world bimanual tasks, ModAR outperforms baselines and continues to improve from human videos.

Limitations

The reported margin is modest: 75% versus 72%, described as an “observed” average success rate, with no confidence intervals or repeated-trial statistics. Per-task numbers, task difficulty distribution, robot hardware and absolute performance are undisclosed, and “highest at all evaluated data scales” comes with no figures attached. Beyond the three bimanual tasks there is no broader generalization, robustness or distribution-shift evaluation. Training from scratch saves compute, but the abstract omits total training scale and data volume, so it is hard to tell whether the conclusion holds only at small scale.

Industry Implications

For teams assembling a world-model pipeline, the most actionable part of this paper is a list of things not to bother doing: piling further RGB reconstruction onto future modalities is likely wasted compute, whereas point tracks, semantic features and depth pay off. The comparison showing parity with a video-model-initialized WAM at one twentieth of the FLOPs and without pretraining strikes at the real cost driver of world models, since video pretraining itself is enormously expensive. If the result replicates at scale, the entry barrier for world-action models drops substantially, which matters most for robotics groups without the resources to pretrain on large video corpora.