EgoLAP: Turning Human Video into Robot Skill via Language Actions

The robot-learning field wants to learn from the internet’s endless human video, but the easier half of that dream is the download and the harder half is the embodiment gap. EgoLAP takes the gap seriously and asks a narrow, productive question: which part of a human action is actually portable to a robot?

Background

Collecting robot demonstrations is slow, expensive, and limited in coverage, which is why egocentric human video looks so attractive: it is abundant and cheap. The trouble is that a human hand and a robot gripper do not share a coordinate system of motion. Human trajectories are embodiment-specific — joint angles, contact points, and dynamics all differ — so simply treating them as supervision for robot control produces a distribution the policy cannot use. The key insight behind EgoLAP is that while the low-level actions are embodiment-specific, the motion intent underneath them is not: the intent carries the task-relevant structure that both humans and robots share, and it is the intent that should be transferred.

Core Innovation

EgoLAP is a vision-language-action pre-training framework that learns from human and robot trajectories jointly through a shared language-based action chain-of-thought. Its central representation expresses motion intent as structured, temporally abstracted language actions — not raw low-level control, and not free-form narration, but an intermediate description of what the motion is trying to achieve. These language actions are paired with motion-level reasoning that is grounded in scene geometry, physics, and object affordances, so the abstraction stays connected to the physical scene it must act on. Because both human and robot data are mapped into the same language-action space, the two sources can be trained together, and the human data provides scale while the robot data anchors the representation to executable control.

Results

• EgoLAP transfers human experience to robot control more effectively than alternative action representations.
• It reaches 80.1% mean real-world task progress, a 2.3x performance gain over alternative action representations.
• Motion-level reasoning outperforms a composite reasoning format that combines subtask, object-box, and visual-trace reasoning.

Limitations

The headline number is “mean task progress,” not success rate, and the two are not interchangeable — progress can be high while a task still fails. The abstract does not report how many real-world tasks or human hours are involved, how the 2.3x gain is distributed across tasks, or how EgoLAP compares against a conventional robot-only pre-training baseline under matched compute; without that comparison it is hard to attribute the gain to human data rather than to the architecture or the reasoning format. The language-action abstraction necessarily discards some fine-grained contact information, and the abstract does not quantify where that loss bites. Variance and failure cases are not reported.

Industry Implications

For teams assembling VLA pre-training data, egocentric human video is the cheapest large-scale source available, and EgoLAP’s message is that the usable signal is the motion intent, not the trajectory. That reframes the data pipeline: instead of chasing ever-more-faithful human-to-robot action mapping, invest in a language-action layer that abstracts intent and can be annotated and reasoned over. The second finding is a caution as well as a tip — motion-level reasoning beat a heavier composite of subtask, object-box and visual-trace supervision, suggesting that more supervision is not automatically better and that the right abstraction may matter more than the volume of auxiliary labels. Any team building an egocentric-data pipeline should treat the language-action representation as the design decision that determines whether that data transfers at all.