Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data

Background

Learning generalizable robot manipulation policies depends on large-scale, diverse demonstration data, and collecting it with robot hardware remains expensive and slow. Egocentric human manipulation videos are rich in scene and task diversity and essentially free at web scale. Prior work showed that retargeting and rendering such videos into robot-format data can yield effective per-task policies at small scale, but whether this approach can provide pretraining benefits for vision-language-action (VLA) models at scale was unexplored.

Core Innovation

Ego2Robot is a scalable pipeline that converts egocentric human manipulation videos into robot training data through three stages: action retargeting, robot-arm visual synthesis, and multi-level quality curation. It supports both curated datasets and in-the-wild videos, producing 18,561 hours of robot training data spanning 15 robot morphologies — the largest ego-to-robot dataset to date. To measure generalization rigorously, the authors extend RoboTwin2.0 with disentangled perturbation axes covering visual appearance, scene layout, embodiment morphology, and task semantics.

Results

Joint pretraining on Ego2Robot-synthesized and robot data consistently improves out-of-distribution generalization across multiple perturbation types. The gains are validated not only in simulation but also on real-robot deployment, indicating that egocentric human video can serve as a practical pretraining source for VLA policies rather than merely a small-scale data trick.

Limitations

Synthesized data remains bounded by the quality of the underlying retargeting and visual synthesis, and the multi-level curation cannot fully remove artifacts from in-the-wild footage. The evaluation centers on manipulation tasks, so conclusions may not transfer directly to other embodiment classes. The scale of compute required to synthesize and curate 18,561 hours of data is not detailed, which matters for teams assessing cost.

Industry Implications

Egocentric human video is one of the largest untapped sources of manipulation demonstrations, and pipelines that convert it into robot-format training data could relieve the teleoperation data bottleneck that currently limits VLA pretraining. For companies building generalist manipulation models, this work points to a concrete path: synthesize first, then fine-tune on a smaller set of real robot data, rather than collecting everything on hardware. The demonstrated real-robot gains strengthen the case for data-synthesis-first strategies in commercial robot learning stacks.