Reconstruct, Practice, Go Real: Guided Self-Improvement for Embodied Agents

· Editorial Team estimated
self-improvement embodied-agents skill-library llm-agent sim-to-real manipulation

Building reliable robot competence across many tasks usually costs a great deal of human effort spent maintaining skills, designing rewards and wiring perception to control. RPG (Reconstruct, Practice, Go Real) is a framework that improves a robot execution system autonomously without updating any model weights. It identifies manipulation capabilities already latent in an offline dataset and constructs related practice tasks in simulation; during practice it diagnoses failures from execution feedback, privileged simulator state and available dataset videos, then develops new reusable symbolic skills, refines existing ones and revises its own system prompt. Cross-task evaluation admits each candidate change, and merged revisions, only after they pass. At test time a multimodal LLM uses the resulting system prompt and skill library to coordinate perception and robot control. On held-out initializations of 22 manipulation tasks, success rises from 28.6% after the first practice round to 95.0% after fifteen, beating every evaluated baseline including ASPIRE (75.5%) and GPT-6 Astra Pro-driven CaP-Agent0 (60.0%). After a common calibration and hardware-adaptation procedure, the frozen system succeeds in all 30 physical trials, ten each on three tasks.

Paper · arXiv:2610.02204

Most robot systems only get better when someone retrains them. RPG asks a different question: can a system reconstruct what it already knows, invent its own practice, and refuse to keep a change until it has earned it?

Background

Giving a robot reliable competence across many tasks is expensive in human labor. Skills have to be written and maintained, rewards have to be designed, and perception has to be integrated with control. Each of these steps normally requires an engineer in the loop, and the result is brittle when the task distribution shifts. A system that could improve its own execution stack from data it already has, without touching model weights, would lower that cost substantially and let the same frozen model cover more of the long tail of manipulation tasks.

Core Innovation

RPG structures autonomous improvement around three moves. It first reconstructs: it scans an offline manipulation dataset to identify capabilities the system can already express, and builds related practice tasks in simulation around them. It then practices: using execution feedback, privileged simulator state and available dataset videos, it diagnoses why attempts fail, and from those diagnoses it develops new reusable symbolic skills, refines existing skills, and revises the system prompt that governs how its skills are selected and sequenced. The critical discipline is cross-task evaluation — before any candidate change or merged revision is retained, it is tested across tasks, so the system accumulates only changes that generalize rather than overfitting to the practice scenario. At test time, a multimodal LLM consumes the evolved system prompt and skill library and coordinates perception with robot control, but the underlying models remain frozen throughout.

Results

  • On held-out initializations of 22 manipulation tasks, task success improves from 28.6% after the first practice round to 95.0% after 15 rounds.
  • It outperforms all evaluated baselines, including ASPIRE (75.5%) and CaP-Agent0 powered by GPT-6 Astra Pro (60.0%).
  • After a common calibration and hardware-adaptation procedure, the frozen system succeeds in all 30 physical trials, ten each across three tasks.
  • Skill construction, skill refinement and prompt revision all occur without updating model weights.

Limitations

The improvement loop lives in simulation: practice, diagnosis and gating all depend on privileged simulator state that does not exist on hardware, so the fidelity of the simulator is a first-order dependency on how far the reported gains transfer. The physical result is real but bounded — 30 trials over three tasks, and only after an explicit calibration and hardware-adaptation step, so the headline “all 30 physical trials” should be read together with that procedure rather than as plug-and-play transfer. The evaluation spans 22 simulated tasks plus three physical ones; the abstract does not report failure cases, compute cost, or how the symbolic skill abstraction copes with continuous, contact-rich control that resists discretization. Whether the system keeps improving, plateaus, or degrades when practice tasks are mismatched to deployment is not characterized.

Industry Implications

The interesting claim for practitioners is not any single number but the shape of the loop: a frozen model plus an evolving skill library and prompt is a direct challenge to the instinct that every capability gain requires a fine-tuning run. That reframes where improvement budgets go — toward simulator fidelity, dataset curation and a rigorous cross-task evaluation harness rather than GPU hours for weight updates. Teams already sitting on large offline manipulation datasets should note that RPG treats those datasets as raw material for self-authored practice rather than as passive training inputs. The dependence on privileged simulation state and on a hardware-adaptation procedure is also the honest cost line: the more the loop leans on simulator truth, the more the sim-to-real gap becomes the ceiling on what the loop can discover.