Teaching a Humanoid to Predict How Body and Terrain Change Together

A legged robot climbing stairs is doing something subtler than it looks. The relevance of a patch of terrain is not a property of the depth image alone; it depends on where the robot’s body is, which foot is loaded, how fast it is moving, and what it is about to do. The same 20-centimetre step is a foothold, an obstacle or irrelevant, depending on all of that. So the useful learning problem is not “fuse vision and proprioception” but “learn the joint state of body and terrain, and learn how actions change it”.

Existing work has attacked the first half well. Terrain-aware sensory fusion and cross-modal prediction have improved how depth and proprioception are combined and related. But feature aggregation does not specify which cross-modal relationships the policy should learn, and predicting one modality from another at the same instant does not constrain how the pair evolves under an action. A policy can have an excellent fused representation and still not know what its next footfall will do.

Core innovation

A compact policy state built by learned queries. Multiple learned queries aggregate depth tokens and proprioceptive tokens into a small set of features that form the policy state. This is a bottleneck by design: the controller should not need the full depth image, only the part of it that bears on the next decision.

Bidirectional masked cross-modal prediction. During training, one modality is masked and the visible modality is asked to predict its latent features, in both directions. Predicting proprioceptive latents from depth forces the visual representation to encode body-relevant geometry; predicting depth latents from proprioception forces the body state to encode the terrain it is standing on. This is a stronger supervisory signal than contrastive alignment because it specifies exactly which relationship must be recoverable.

An action-conditioned predictor of the next fused state. The third component maps the current policy state plus the executed action to the next fused state, with an exponential-moving-average target defined directly in the policy representation space. This is the piece that turns a representation into something usable for control: it teaches the policy what its own actions do to the body-terrain relationship, not just what the relationship currently is.

The whole thing is trained in one stage, which matters practically — no separate world-model pretraining, no distillation schedule.

Results

In simulation, the framework improves terrain traversal over the fusion alternatives and ablations it is compared against. The more informative experiments are the degradation ones: with depth dropout and with delayed depth, both predictive objectives help, which is what you would expect if the policy has learned to lean on proprioceptive evidence about terrain when vision is unavailable or stale.

The hardware result is the headline. On a 23-DoF Noetix E1 using onboard sensing, the policy succeeded in 98 of 100 trials spanning stairs, a 40 cm platform, a 70 cm gap, stepping stones, and backward stair traversal without rearward terrain visibility. That last case is the most telling: descending stairs backwards means the terrain the robot is stepping onto is not in its depth sensor’s field of view at all. Success there is only possible if proprioception and the learned dynamics prior are carrying real information about what the feet will meet.

Limitations

The evidence comes from one robot platform and one policy. Ninety-eight out of one hundred trials is strong for a research paper, but it is a small sample for any safety argument, and the trials are aggregated across obstacle types rather than reported per condition with confidence intervals.

There is no quantified sim-to-real analysis: the paper reports that the policy transferred without real-world fine-tuning, but not how much performance was lost relative to simulation, nor which conditions degraded most. For a method whose selling point is a learned dynamics prior, the size of the simulation gap is exactly what a reader needs.

The approach also assumes depth sensing is available and reasonably calibrated. The depth-dropout experiments show graceful degradation, but a robot whose camera is blocked or whose depth scale is wrong is outside the tested envelope.

Industry implications

Humanoid locomotion has moved in eighteen months from “can it walk without falling” to “can it do useful work in a building”. The binding constraint has shifted accordingly, from balance to reliability on the specific terrain a customer site happens to have — loading docks, mezzanines with open edges, stairs with inconsistent riser heights.

A policy that explicitly models how its actions change the body-terrain state is a better starting point for that reliability work than one that only fuses sensors, because the failure modes it can anticipate are the ones that involve its own motion. It also produces a representation that is easier to monitor: if the predicted next state and the observed one diverge, that is a signal something is wrong, which is the raw material for a runtime safety check.

The commercial caveat is validation cost. Ninety-eight of a hundred trials on five obstacle classes is a demonstration, not a safety case. Deploying a humanoid near people requires per-site terrain characterisation and far larger trial counts, and the paper’s method does not remove that requirement — it only makes the policy more likely to survive it.