World In Your Hands: A Thousand Hours of In-the-Wild Manipulation Data, Made Open

· Editorial Team estimated
manipulation datasets data-collection human-to-robot-transfer cross-embodiment open-source

Robot manipulation is gated less by policy architecture than by data, and World In Your Hands (WIYH) attacks that constraint at the ecosystem level: a wearable capture kit with an automatic labelling pipeline, a dataset of more than 1,000 hours of in-the-wild multimodal manipulation spanning hundreds of skills, and annotations plus benchmarks that reach from perception to action. The authors report millimetre-scale motion accuracy from the capture hardware and show that folding WIYH's human data into robot training lifts manipulation success in cluttered scenes from 8% to 60%. Both the dataset and the hardware designs are released openly, positioning WIYH as infrastructure for cross-embodiment policy learning rather than a single-task result.

Paper · arXiv:2512.24310

Robot manipulation has a data problem that no amount of policy cleverness has solved. Every lab can collect demonstrations on its own hardware, but the process does not scale: teleoperation is slow, expensive, and tied to the specific arm in the room, while the abundant alternative — egocentric human video — arrives without action labels and with a body that does not match the robot. World In Your Hands (WIYH) is an attempt to route around both limits by industrialising the collection of human manipulation data in the wild and then giving it away.

Background

The standard recipe for teaching a manipulation policy is to record expert behaviour, usually through teleoperation, and train on it. That recipe works, but its cost is measured in operator hours per demonstration, and the demonstrations are bound to the embodiment that produced them. Human video offers far more coverage of everyday interaction, yet it lacks the action supervision a controller needs and reflects human kinematics and appearance rather than a gripper’s. The result is a field that keeps rebuilding small, platform-specific datasets instead of compounding a shared one. WIYH starts from the premise that the scarce resource is not algorithms but labelled, physically accurate human manipulation.

Core Innovation

WIYH is presented as an ecosystem rather than a dataset, with three coupled parts:

  • The Oracle Suite. A wearable data collection kit paired with an automatic labelling pipeline for accurate motion capture, so demonstrations can be gathered wherever people already work rather than in a motion-capture room.
  • The WIYH Dataset. More than 1,000 hours of multimodal manipulation data collected in the wild, covering hundreds of skills across diverse real-world scenarios.
  • Annotations and benchmarks. Extensive labels that support tasks spanning perception through to action, so the data can be used to train and to evaluate, not merely to pre-train.

The design choice that matters most is openness: both the data and the hardware design are released. A capture kit is only useful to the community if others can build it, and a labelling pipeline is only useful if others can run it on their own footage.

Results

  • Scale and accuracy. Over 1,000 hours of in-the-wild manipulation with millimetre-scale motion accuracy, across hundreds of skills.
  • Transfer to robots. Integrating WIYH’s human-centric data improves robotic manipulation success in cluttered scenes from 8% to 60%.
  • Support for the full pipeline. The annotations and benchmarks are described as covering tasks from perception to action, and all data and hardware designs are to be open-sourced.

The transfer number is the one to watch. An eight-fold improvement in a cluttered setting suggests that the value of the data lies specifically in the situations where policies usually fail, rather than in the easy, well-lit demonstrations that dominate existing corpora.

Limitations

The 8% to 60% result is reported for a single evaluation setting, and the abstract gives no breakdown by skill category, embodiment, or scene type, so it is hard to tell how broadly the gain generalises. The millimetre-scale figure describes capture accuracy; how much labelling error the automatic pipeline adds, and how that error propagates into policy performance, is not stated. Human-to-robot transfer still relies on resolving the embodiment gap, and the abstract does not enumerate the robot platforms or morphologies the data has been validated on. Collecting in the wild buys diversity but also noise, and the abstract does not describe how variability, failed attempts, or unlabelled segments are curated. Finally, the promise that everything is open-source is forward-looking at the time of writing, so its practical value depends on the released artefacts being complete and documented.

Industry Implications

For teams building manipulation systems, the most immediate consequence is economic. If a shared, open corpus of human manipulation can substitute for a meaningful fraction of in-house teleoperation, it removes one of the largest recurring costs in robot learning and lowers the barrier for smaller labs and startups that cannot afford a teleoperation fleet. Releasing the capture hardware amplifies that effect: a kit other people can build turns a dataset into a pipeline, and pipelines are what compound. The strategic implication is that human data is becoming a scaling axis for generalist policies, and whoever owns the collection and annotation stack owns a durable advantage. The open question is coverage — whether a corpus gathered by one group generalises to the tasks and embodiments that commercial deployments actually require. Until that is demonstrated across platforms and skills, WIYH should be read as powerful infrastructure with a strong first result, not as a solved data problem.