PRISM: Turning a Handful of Real Videos into Scalable Humanoid Loco-Manipulation

· Editorial Team estimated
humanoid loco-manipulation imitation-learning video-generation sim-to-real manipulation

Teaching a humanoid to carry objects by watching people is limited less by policy design than by data: useful clips must show a whole body and an unoccluded interaction with the object, and such footage is scarce. PRISM is a real-to-sim-to-real framework that amplifies a handful of real videos into a large, diverse training set. It generates hundreds of counterfactual human-object interaction videos with video-to-video generation, then uses a contact-anchored real-to-sim pipeline to reconstruct human and object motion and retarget the imperfect video into physically plausible trajectories. The intra-class variability across these videos lets a single policy generalize to unseen objects within each category. Deployed on a real humanoid with no real-world fine-tuning, using only onboard depth, the policy picks up, carries and drops boxes, barrels, bins and balls across novel instances, sizes and initial configurations.

Paper · arXiv:2609.38172

Building a humanoid that can carry a box, a barrel or a bin on request is a data problem before it is a control problem. The videos that would teach it are rare, because they must capture a person’s whole body and an unobstructed interaction with the object. PRISM asks whether a few good recordings can be amplified into a training set large enough to generalize.

Background

Visual imitation is one of the most promising routes to generalist humanoids that can both move and manipulate, but it runs into a practical wall. The demonstrations it needs are demanding to film: clips have to show the person’s full body and an unoccluded interaction with the object, at a level of diversity that no small hand-collected set provides. Record the wrong camera angle and the hand-object contact is hidden; record one example and the policy overfits to that single object and configuration. Scaling loco-manipulation therefore stalls on acquisition of high-quality interaction video rather than on model capacity.

Core Innovation

PRISM is a real-to-sim-to-real framework built to break that data bottleneck. Its first stage uses video-to-video (V2V) generation to turn a few exemplar real videos into hundreds of diverse “counterfactual” human-object interaction videos, expanding a single demonstration into a family of plausible variations. The second stage is a contact-anchored real-to-sim pipeline that reconstructs both human and object motion and retargets this generated, imperfect video data into physically plausible trajectories. The deliberate payoff is intra-class variability: because the counterfactual set varies object instance, appearance and configuration within a category, a single policy trained on it generalizes to unseen objects of the same category instead of memorizing one clip. The pipeline is then closed back to reality by deploying the trained policy on a real robot.

Results

  • The full pipeline is demonstrated on a real humanoid with no real-world fine-tuning at all.
  • Using only onboard depth observations, the humanoid picks up, carries and drops objects including boxes, barrels, bins and balls.
  • The policy succeeds across novel instances, sizes and initial configurations of those objects.
  • Training data is generated by amplifying a handful of real videos into hundreds of counterfactual human-object interaction videos, which are then retargeted into physically plausible trajectories.

Limitations

The evidence is primarily behavioural rather than a broad quantitative benchmark: the paper demonstrates the pipeline across a set of object categories and reports that the capability works, without a large-scale success-rate table across many tasks or scenes. Generalization is explicitly scoped to unseen objects within each category, so cross-category or open-set generalization is not established. Relying only on onboard depth is a deployment strength but also caps how much perception can disambiguate difficult scenes. Finally, because there is no real-world fine-tuning, any artifact introduced by V2V generation or by the retargeting step is absorbed directly by the policy, so the quality ceiling is set by the synthetic pipeline itself.

Industry Implications

For humanoid developers the important claim is economic: if a few real clips plus generative amplification plus physics-grounded retargeting are enough to bootstrap carrying behaviour, the cost of a new loco-manipulation skill shifts from filming at scale to engineering and running a pipeline. That is a far more scalable unit of work, and it is exactly the trade a company wants when it needs to add skills across many object types. Operating from onboard depth alone also matters for bill-of-materials and form factor, since it avoids external motion-capture or dense sensing during deployment. The open question for product teams is coverage: category-scoped generalization is useful, but real deployments present objects that do not fit neatly into the categories the pipeline was built around.