DexRoam: Learning Mobile Bimanual Dexterous Manipulation from Egocentric Whole-Body Human Demonstrations

Robot demonstrations are the bottleneck for mobile dexterous manipulation. This paper asks what happens if you capture the demonstrations on a human body instead - without throwing away the fine-grained whole-body coupling that makes the skill work.

Background

Mobile bimanual dexterous manipulation is one of the hardest data problems in robotics: a single trajectory has to coordinate locomotion, whole-body posture and finger-level dexterity, and today every hour of that data must be produced on a real robot. Egocentric human demonstrations look like the obvious scalable substitute, but most prior pipelines make the human-to-robot transfer easier by simplifying the motion, discarding precisely the fine-grained, coupled structure such tasks depend on. The result is training data that is cheap but no longer really describes the skill.

Core Innovation

DexRoam keeps whole-body motion continuous and coupled all the way through the transfer. To collect demonstrations at scale it uses a tracker-free capture system: a consumer VR headset plus a head-mounted stereo camera, requiring no external cameras or motion trackers. Three explicit alignment stages then map the captured motion into the robot action space - embodiment alignment (human body to robot morphology), action-semantic alignment (human motion to robot actions) and temporal alignment (timing between the two). Because the fine-grained whole-body structure survives this mapping, human and robot demonstrations can be jointly learned by standard VLA policies rather than special-cased.

Results

• On GR00T N1.7, adding human demonstrations raises average real-world success from 29% to 56%.
• On pi0.5, average success rises from 32% to 57%.
• Robot-only performance can be matched using only half the robot demonstrations.
• Ablations confirm that each of the three alignment stages is necessary.

Limitations

The measured outcome is still roughly half of attempts succeeding, so this is a large data-efficiency gain rather than a solved policy. The ceiling depends heavily on the quality of the tracker-free capture, which uses only a VR headset and a head-mounted stereo camera; the abstract does not report how robust the pipeline is to noisy or failed captures, nor how well the alignment stages hold up on embodiments and task families outside those tested. The claim of consistent improvement is made for “different VLA backbones,” but only two are named with numbers, so breadth beyond them is not yet established.

Industry Implications

The economics of manipulation data are the point. If a VR headset and a head-mounted camera can produce demonstrations that let a standard VLA policy learn mobile bimanual dexterity, then scaling robot skills stops being gated on robot time and becomes gated on human time, which is far cheaper and more available. The result that half the robot demonstrations suffice for equal performance is the concrete lever: it directly cuts the cost of every data-collection campaign. For teams building humanoid or mobile-manipulation fleets, the operational question shifts from how many robots to tie up collecting data to how to instrument humans well.