Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning

· Editorial Team estimated
robot-learning behavior-cloning reinforcement-learning bimanual-manipulation self-improvement

Q-Planning equips a large visuomotor behavior-cloning policy with a small off-policy Q-function, enabling value-guided action selection at inference and online self-improvement that fine-tunes only the Q-function. On LIBERO and bimanual RoboTwin, ten self-improvement iterations lift every benchmark score (LIBERO-10 93% to 99%, RoboTwin 83.8% to 91.4%); on two contact-rich bimanual real-robot tasks the same loop improves purely from its own deployment rollouts (stack-cups 40% to 90%, insert-wallet 25% to 80%), whereas filtered SFT on successful rollouts stalls at 55% and 30%.

Paper · arXiv:2608.21204

Behavior cloning (BC) has driven much of the recent progress in robot manipulation, but it cannot learn from its own mistakes: a policy that fails is stuck unless a human provides more demonstrations. Reinforcement learning fine-tuning would offer a path to self-improvement, yet it has proven extremely difficult to scale to the multi-billion-parameter models behind modern robot policies. Q-Planning attacks this asymmetry directly.

Core Innovation

The key insight is that a Q-function estimates value rather than imitating actions, so it can be trained on the same successful demonstrations as the BC policy and later absorb both successful and failed deployment rollouts — something BC cannot do. Q-Planning therefore equips a large visuomotor BC policy with a small off-policy Q-function and exploits this asymmetry in two ways:

  • Value-guided action selection at inference: a single-step Q-weighted average over BC action draws, replacing naive imitation with value-aware choice.
  • Online self-improvement: only the Q-function is fine-tuned on new rollouts; the BC weights stay frozen, avoiding catastrophic forgetting and sidestepping the instability of full-model RL fine-tuning.

Because no auxiliary actor is trained, the method stays cheap and stable even when the underlying policy is huge.

Results

  • LIBERO-10 rises from 93% to 99% over ten self-improvement iterations; RoboTwin from 83.8% to 91.4%.
  • Successful episodes shorten on the near-ceiling suites (LIBERO-Object, LIBERO-Goal).
  • Real bimanual robot, no human intervention: stack-cups 40% to 90% and insert-wallet 25% to 80% in five iterations, while SFT on successful rollouts alone stalls at 55% and 30%.
  • Under an identical online budget, Q-Planning is the only method among Best-of-N, filtered SFT, IBRL, DSRL, and DAWR that improves stably from failures.

Limitations

The real-world evidence covers two contact-rich bimanual tasks, and the simulation evidence is confined to LIBERO and RoboTwin suites; breadth across many task families is not yet demonstrated. The single-step Q-weighted selection is a pragmatic approximation rather than a full lookahead, and the abstract does not analyze how Q-estimation error grows over many deployment iterations. The approach also inherits the base BC policy’s dependence on demonstration quality.

Industry Implications

Self-improvement from a robot’s own deployment data — without extra human demonstrations — directly attacks the largest cost in modern robot learning: data collection. For teams shipping VLA-style manipulation products in warehouses, assembly, or home settings, a loop that fine-tunes only a small Q-function is practical to run on fleet hardware, and value-guided selection can be switched on without retraining the policy. If the result holds across more tasks, it points toward a credible production recipe for continuous robot improvement.