InterEvolve: Test-Time Evolution of Reward Programs for Humanoid Loco-Manipulation

A humanoid controller rarely needs to be taught a new task from scratch — it usually already holds the motor competence, and the real problem is writing a reward that can reach it. InterEvolve makes that reward something a machine can rewrite.

Background

Humanoid loco-manipulation is hard to scale because each new task has traditionally meant retraining a controller, and reward engineering is the bottleneck inside that loop: human-written rewards are brittle, task-specific and rarely light up the full competence of a broad controller. The paper’s premise is that a general controller already contains much of what a new task needs, and that this latent competence becomes accessible only through an interface between planning and control that is expressive enough to specify contact-rich, multi-stage interactions yet executable and measurable enough for execution feedback to steer planning.

Core Innovation

InterEvolve builds that interface from two components. The first is an object-aware forward-backward (FB) behavioral foundation model: object residuals layered onto a frozen body prior let a newly specified reward about the body or the objects be turned into loco-manipulation behavior at test time, so the controller can be repurposed without retraining. The second component expresses tasks as reward programs — staged rewards with explicit completion conditions and tunable constants. An LLM agent revises the program structure in context, guided by execution feedback and a skill library of already-verified programs, while a numerical optimizer tunes the constants. Crucially, every candidate is verified across parallel simulation scenarios before it is admitted, so the search explores novel ways to induce, repurpose and compose the controller’s existing motor competence rather than drifting into unverifiable attempts.

Results

• Human-designed rewards leave much of the FB model’s loco-manipulation competence untapped, whereas the evolved programs release it, sometimes through novel strategies.
• InterEvolve produces behaviors for diverse tasks, complex scenes and long-horizon compositions in simulation.
• Evolved skills run autonomously on a physical Unitree G1, driven by egocentric onboard perception.
• Verification of every candidate across parallel simulation scenarios is central to the search’s reliability.

Limitations

The reported evidence is qualitative in the abstract: gains are described as releasing untapped competence and producing behaviors for diverse tasks, without head-to-head numbers, success rates or a baseline table, so the magnitude of improvement over human-designed rewards is not quantified here. The loop is anchored in simulation, and despite onboard execution on a real G1, the abstract does not report how many physical trials were run or how the evolved programs behave when the simulator diverges from hardware. Scaling behavior with task complexity, the cost of verifying candidates across parallel scenarios, and the risk that LLM-revised reward structure encodes simulator artifacts rather than task intent are all open.

Industry Implications

For teams working on humanoid platforms, the reframing is the payload: if a broad controller already holds the competence and the bottleneck is the reward, then the leverage moves from retraining pipelines to reward-program search and verification infrastructure. Treating reward structure as an editable artifact that an LLM proposes and a simulator gates is a plausible path to the long tail of tasks that are too small or too varied to justify custom training runs. The emphasis on verifying every candidate across parallel scenarios also points at a practical maturity requirement — the value of this kind of test-time evolution scales directly with simulation throughput and with how faithfully the sim preserves contact. Unitree G1 owners should read the physical result as a strong signal that the interface is deployable, while noting that the abstract leaves the sim-to-real accounting to future work.