Generate, Track, Improve: Perceptive Multi-Skill Humanoid Locomotion with RL-Fine-Tuned Motion Generators

· Editorial Team estimated
humanoid locomotion reinforcement-learning motion-generation depth-perception sim-to-real

A general-purpose humanoid needs one controller that can walk, run, jump and climb stairs over whatever terrain it meets, without leaning on odometry or a prebuilt height map. This work splits the job into two layers: a perceptive flow-matching motion generator that plans whole-body trajectories directly from raw depth images, and a perceptive tracking policy trained with control-guided reinforcement learning that follows those motions. Both are trained on terrain-consistent motion clips derived from dynamically optimized human data. The central contribution is an off-policy RL fine-tuning loop that improves the generator itself, using structured search to gather data for advantage-weighted regression - far more sample-efficient than fine-tuning the tracker with on-policy residuals. Terrain traversals improve by up to 25 percentage points and skill selection by up to 80 percentage points, and a single policy pair drives a Unitree G1 to walk, run, stand, jump on and off boxes and climb outdoor stairs.

Paper · arXiv:2609.31577

A two-layer humanoid controller that plans motion from depth and then improves its own motion generator, instead of patching a tracker forever.

Background

The ambition for general-purpose humanoids is a locomotion controller that is multi-skill, perceptive, dynamic and robust enough to go anywhere a person can. In practice, controllers tend to be good at one of those things. Perceptive policies often rely on odometry or a prebuilt height map, which is fragile outdoors and awkward to maintain; motion generators trained on motion-capture data produce natural but terrain-inconsistent trajectories; and when the tracker fails, the usual fix is to bolt on a residual policy that fights the reference rather than correcting it.

Core Innovation

The architecture is deliberately two-layer. A perceptive flow-matching motion generator plans whole-body trajectories from raw depth images; a perceptive tracking policy, trained with control-guided reinforcement learning, follows those trajectories. Both are trained on a library of terrain-consistent motion clips built from dynamically optimized human data, which yields references that are both accurate in velocity tracking and consistent with the terrain.

The paper’s central contribution is what happens after that: a simple off-policy RL fine-tuning loop that improves the generator using a structured search method to gather data for advantage-weighted regression. Because the loop is off-policy, it is far more sample-efficient than on-policy residual fine-tuning, and it improves terrain consistency on unseen geometries and unseen skill compositions rather than memorizing the training terrains. Because perception operates on raw depth, no odometry or height map is required, and two cameras let the policy see terrain approaching from further away and adjust its velocity regardless of the commanded speed.

Results

  • Successful terrain traversals increased by up to 25 percentage points and skill selection improved by up to 80 percentage points.
  • The off-policy fine-tuning loop is reported to be much more sample-efficient than on-policy residual fine-tuning.
  • Terrain consistency improves on unseen geometries and skill compositions.
  • A single policy pair enables a Unitree G1 to walk, run, stand, jump on and off boxes and traverse stairs in outdoor environments.
  • No odometry or height maps are used; raw depth images and two cameras suffice for outdoor deployment.

Limitations

The abstract reports relative improvements only - up to 25 and 80 percentage points - without absolute success rates, so the baseline level being improved is unknown; an 80-point gain in “skill selection” in particular needs the metric defined before it can be weighed. All evidence comes from a single humanoid platform (Unitree G1), and depth-only perception has well-known failure modes - reflective, transparent or textureless surfaces, direct sunlight - that outdoor claims invite but the abstract does not address. The reported gains also concern the generator’s fine-tuning loop; how the tracker behaves when the generator proposes references far outside its training distribution is not described.

Industry Implications

Two things here are directly useful to anyone shipping a legged platform. First, dropping odometry and height maps in favour of raw depth is an operational simplification as much as an algorithmic one: fewer state estimators to calibrate, fewer assumptions to violate when the robot leaves the lab, and no map to keep synchronized with a changing site. Second, fixing the generator instead of adding a residual tracker policy is a maintenance argument - a single improved reference model is easier to test and audit than a generator plus an ever-growing corrective layer - and the off-policy loop’s sample efficiency matters economically, because on-robot data collection is the expensive part of legged control. The single-policy-pair result, covering walking, running, jumping on and off boxes and stairs, is the shape of controller operators actually want: one artifact, many behaviours, no task-specific retraining.