ViT-H: Generating Humanoid Motion from Video, Then Making It Executable

· Editorial Team estimated
humanoid whole-body control diffusion models video-to-motion preference optimization

Learning whole-body humanoid control from monocular human video is a scalable route to diverse behaviours, but reconstruction-retargeting-tracking pipelines optimise each stage separately, accumulate error between stages, and cannot use downstream tracking feedback to make reference motions more executable. ViT-H replaces them with a generation-and-tracking paradigm: a diffusion model emits humanoid reference motion directly from video, bypassing reconstruction and retargeting, and is then post-trained with direct preference optimisation on tracking outcomes. An autoregressive variant enables real-time video teleoperation below 400 ms. On a Unitree G1, ViT-H beats state-of-the-art pipelines on both motion quality and executability, and scales with model and data size.

Paper · arXiv:2610.32966

Humanoid behaviour diversity is limited by data. A motion-capture studio produces excellent reference motion, but its cost and footprint mean it will never cover the distribution of everyday movement. Monocular human video on the internet is effectively unbounded; the open question is how to turn it into joint trajectories a robot can execute.

The standard answer is a three-stage pipeline: estimate human motion from video, retarget the human skeleton onto the humanoid’s linkage, then train a tracking policy to follow the resulting reference. Each stage optimises its own objective. Reconstruction aims to look human, retargeting aims for kinematic feasibility, and only at the tracking stage does it become apparent that some references simply cannot be followed. Error compounds across the seams, and the tracking failure signal has no path back to the reference motion that caused it.

Core innovation

ViT-H compresses the chain into two stages: generate, then track.

Direct humanoid reference generation. A diffusion model maps video straight to humanoid reference motion, skipping both human motion reconstruction and retargeting. There is no intermediate representation left to align.

Preference optimisation from tracking outcomes. Generated references are handed to a motion tracker for execution, and tracking success and quality become preference signals used to post-train the generator with direct preference optimisation. This is the load-bearing idea: references are no longer optimised for resembling the person in the video, but for executability downstream.

An autoregressive real-time variant. A streaming version converts video frame by frame into humanoid motion with sub-400 ms latency, enabling live video teleoperation.

Results

  • On a Unitree G1, ViT-H outperforms state-of-the-art reconstruction-retargeting-tracking pipelines on both motion quality and executability.
  • Performance scales with model size and with both pre-training and post-training data.
  • Tracking feedback improves executability consistently across different motion trackers, indicating the gain comes from better reference motion rather than overfitting to one tracker.
  • The autoregressive variant supports real-time teleoperation below 400 ms.

Limitations

The abstract reports relative conclusions without motion-quality metrics, tracking error values or failure case analysis, so the size of the improvement cannot be judged independently. The method needs a working tracker to supply preference signal and risks overfitting that tracker’s preferences. Evaluation is confined to a single Unitree G1 platform, leaving cross-embodiment transfer unknown, and the latency claim is not validated under real network and perception noise.

Industry implications

If video can be converted directly into executable humanoid motion, the source of motion data expands from mocap studios to the entire internet video corpus, which is a structural change to the data cost base of any humanoid vendor. The more interesting piece is the DPO loop: it transfers the definition of a good reference from visual similarity to the robot’s own executability, precisely the feedback path three-stage pipelines lack. Practically, generator and tracker can now evolve independently, and the real-time variant offers a route to low-cost teleoperation and remote demonstration that needs no motion-capture suit.