In-Hand Dexterity That Survives Contact: Pose Reaching Across 4,800 Real Goals

Ask anyone who has worked with a multi-fingered hand what the hard part is, and the answer is rarely the grasp. Grasping is a solved-enough problem: close the fingers on an object and it usually stays. What remains open is everything after the grasp — moving that object to a new position and orientation inside the hand itself, without letting go, without the fingers losing contact, and without the object escaping when the wrist turns.

This is called in-hand object pose reaching, and it is a much harsher task than the rotation demos that populate dexterous manipulation videos. The goal is not a fixed spin but an arbitrary target pose: almost any position and orientation reachable by the fingers, at any wrist orientation, chained into long sequences of consecutive goals. Each transition raises the chance of a drop, and the object’s mass distribution changes the finger forces required at every step.

The usual bottleneck is data. Human demonstration of in-hand repositioning is close to impossible to collect at scale — a person cannot teleoperate a five-fingered hand through thousands of consecutive regrasps — and simulation-trained policies have historically failed to survive the transfer to real contact dynamics.

Core innovation

The authors treat the problem as a systems problem rather than a policy-architecture problem, and they are explicit that no human data is used anywhere in training.

Goal and reset diversity as the main lever. Rather than sampling a narrow band of object poses, the training distribution is built from generated grasp datasets, so the policy sees a wide spread of initial finger configurations and goal poses. Diversity of resets is what stops the policy from overfitting to a handful of comfortable grasps.

An adaptive friction curriculum. Contact friction is the single parameter that most reliably breaks sim-to-real transfer for a hand. Instead of a fixed friction value, the curriculum is adapted during training, so the policy is forced to remain competent across a range of friction conditions rather than tuned to one.

Careful system identification. The gap between simulated and physical finger dynamics is closed by identifying the real platform’s parameters, which is what makes zero-shot deployment plausible instead of aspirational.

A learned residual safety filter. On top of the task policy sits a second learned component that corrects the commanded action with adjustable conservativeness. This is the part that turns a capable policy into a usable one: an operator can dial the balance between how many goals are reached and how often the object is dropped, without retraining.

Results

The evaluation is unusually thorough for this literature: 13.3 hours of real-robot testing, 4,800 unseen goals, four objects, and diverse wrist orientations.

• The state-based policy reaches 95.2% of goals within 2 cm and 0.4 rad.
• With the safety filter engaged, drops fall by 75% while 91.5% of goals are still reached — a 3.7 point sacrifice in success for a three-quarter cut in failures.
• Deployment works with either motion capture or a single wrist-mounted depth camera, so the perception requirement is one sensor on the hand rather than an external tracking rig.
• Beyond the benchmark, the policies spin a lightbulb held in an arbitrary pose, reach goals outside the training distribution, and withstand fast arm motion and external perturbations.

The camera-only configuration is the commercially interesting one. Because the policy needs nothing except a wrist view, it becomes a portable skill: the authors demonstrate screwing in a lightbulb, which is a task defined by continuous in-hand adjustment rather than by a single grasp.

Limitations

The headline 95.2% belongs to the state-based policy; the wrist-camera variant is demonstrated qualitatively as a portable skill rather than reported with its own success and drop statistics, so the perception cost of dropping motion capture is not quantified. Four objects is a narrow object set, and the abstract does not state their geometry or mass range, which matters because in-hand pose reaching is highly sensitive to both. No comparison is given against a policy trained with domain randomization instead of identified system parameters, so the individual contribution of the identification step is not isolated. The safety filter reduces drops but does not eliminate them, and no failure taxonomy is reported — it is not clear whether the remaining drops concentrate in particular wrist orientations or near the edge of the reachable workspace.

Industry implications

Two things here are directly transferable to a product program. The first is the evaluation protocol: 4,800 goals over 13.3 hours, reporting both reach accuracy and drop rate, is the kind of reliability data that a dexterous-hand supplier needs before a customer will put the hand near anything valuable. Most published dexterity results report success on curated objects and say nothing about how often the object is fumbled. The second is the filter’s adjustable conservativeness, which gives integrators a single knob to move between throughput and safety per deployment — a warehouse picking line and a lab automation cell do not want the same trade-off, and neither wants to retrain a policy to change it.

The wrist-camera result also matters for system cost. A hand skill that runs from one onboard camera, with no external motion capture and no fixed cell calibration, is a skill that can be moved between robots and workstations. That portability, more than the 95.2%, is what makes in-hand manipulation start to look like a deployable capability rather than a benchmark.