Whole-Body UMI: Turning Hand-Held Manipulation Demos into Humanoid Whole-Body Skills

Humanoid robots are finally good enough at staying upright that the interesting question has shifted to what they should do with their hands — and there the field runs into a familiar wall. Teaching a humanoid to open a drawer or place an object on a shelf normally means teleoperation: an operator in a rig, hours per task, repeated for every new skill. Whole-Body UMI (WB-UMI) asks whether the cheapest data source available for manipulation can be stretched to cover the whole body.

Background

The Universal Manipulation Interface changed the economics of collecting manipulation data. Instead of teleoperating a robot, a person uses a hand-held gripper and camera rig; the resulting recordings are cheap, portable, and directly comparable across sites, which is why UMI-style datasets now underpin a large share of manipulation policy work. The catch is that UMI records where the end effector went. A humanoid does not get to place its hand anywhere it likes: the arm, torso, hips, and legs have to move together, keep balance, and respect joint limits before the fingertip arrives at the recorded pose. End-effector trajectories are therefore an underdetermined specification for a whole-body robot, and that underdetermination is exactly what teleoperation was implicitly solving.

Core Innovation

WB-UMI separates the problem into two learners joined by a narrow interface.

• A shared end-effector interface. Task semantics live on one side, whole-body coordination on the other, and the end-effector is the only thing that has to pass between them.
• A diffusion policy for tasks. The policy is trained on native UMI demonstrations, exactly as it would be for a fixed-base arm.
• A task-agnostic motion generator for the body. WB-UMI itself is an end-effector-conditioned, real-time generator that learns whole-body coordination from retargeted motion capture. It is trained once, independently of any task.
• No paired data required. Because the two halves learn separately, task-specific collection needs neither body trackers nor synchronised image and whole-body motion — a person can record a new task with the UMI rig alone.
• An asynchronous deployment hierarchy. At run time the diffusion policy, the motion generator, and a whole-body controller are integrated asynchronously, with latency compensation and measured-state feedback to keep the layers consistent.

Results

• Real-time closed-loop transfer on a G1 humanoid across four tasks.
• Drawer closing: 90% success.
• Shelf pick-and-place: 80% success.
• Ball toss: 30% success.
• Loco-manipulation pick-and-place: 40% success.

The split in the numbers is informative rather than embarrassing. Static and quasi-static manipulation transfers well, which is precisely the regime where a decoupled interface should work: the body settles, the arm follows the end-effector reference, and coordination is mostly posture. The dynamic tasks are where a single end-effector stream stops being enough, because a throw or a walking-and-carrying manoeuvre depends on momentum, contact timing, and whole-body state in ways an end-effector trajectory does not encode.

Limitations

The generator’s training depends on retargeted motion capture, so a whole-body motion corpus still has to exist somewhere upstream — the paper removes that burden from task-specific collection, not from the pipeline as a whole. Results come from a single humanoid platform, the G1, so cross-platform generality is untested. Success rates on the two dynamic tasks are low enough that they should be read as feasibility evidence, not capability. The abstract reports no latency, compute, or controller-bandwidth figures, yet an asynchronous hierarchy with latency compensation lives or dies on those numbers. Treating the end-effector as the entire task interface is also a modelling choice with a boundary: it excludes tasks where the body itself is the instrument, such as bracing, climbing, or manipulating with a shoulder or torso.

Industry Implications

The practical value here is an order-of-magnitude change in what it costs to add a skill to a humanoid. If a new task can be specified by recording a person with a UMI rig, the marginal cost of expanding a humanoid’s repertoire drops from operator-hours in a teleoperation rig to a few minutes of hand-held demonstration plus one shared motion generator. That is the difference between a humanoid fleet that ships with a fixed task list and one that can be taught by the people who need the work done. It also makes UMI-style data, already the de-facto currency of manipulation research, valuable for humanoid vendors rather than only for arm-based manipulation. The honest caveat is the 30% and 40% on dynamic tasks: whole-body agility remains the hard part, and the frozen-interface design has a visible ceiling there. The near-term opportunity is in the static and quasi-static manipulation that fills warehouses, kitchens, and labs — the boring tasks where 80% to 90% is already useful and where the data-collection economics now clearly favour this approach.