Value as Memory: Online Value-Guided Post-training for General Robot Policies

· Editorial Team estimated
reinforcement-learning online-learning generalist-policies value-function post-training long-horizon

Real-world post-training of generalist robot policies faces two challenges: partial observability makes action selection ambiguous, and offline post-training methods delay feedback from deployment experience. Value as Memory (VaM) addresses both with value caching and online real-world reinforcement learning: value caching conditions actions on recent value predictions as task-progress memory, while online updates combine offline demonstrations, off-policy interventions and on-policy rollouts for sample-efficient adaptation, moving from an offline RL warmup to online autonomous learning with sparse rewards. Across long-horizon tasks with deformable objects and contact-rich manipulation, VaM yields a 29.4% mean absolute success-rate gain from value caching and a 32.8% gain from online autonomous learning over matched offline training; the online schedule performs 8x as many policy iterations under the same gradient-step budget, with over 2x lower recorded time per gradient step.

Paper · arXiv:2610.22368

A generalist policy that looks strong in the lab is only a starting point — the real test is whether it keeps learning once it is deployed.

Background

Real-world post-training of generalist robot policies faces two intertwined problems. First, partial observability makes action selection ambiguous: the same observation often admits several reasonable actions, so the policy must infer task progress from history. Second, offline post-training delays feedback from deployment experience — failures encountered in the real world can only be exploited after a full training round completes, during which the same mistakes recur. Solving both requires a compact state representation of history and a path that turns online experience into policy improvement immediately.

Core Innovation

Value as Memory (VaM) addresses both problems with value caching plus online real-world reinforcement learning. Value caching feeds recent value predictions back into the policy as task-progress memory to condition actions, so the controller “remembers” how far the task has progressed without relying on ever-longer observation histories. For online updates, VaM unifies three data sources: offline demonstrations supply an initial reasonable behavior, off-policy interventions correct clear deployment mistakes, and on-policy rollouts keep exploring and exploiting sparse rewards. Together they form a sample-efficient adaptation loop that moves smoothly from offline RL warmup to online autonomous learning. On the engineering side, data collection and training run concurrently to cut waiting time.

Results

  • Across long-horizon tasks with deformable objects and contact-rich manipulation, value caching yields a 29.4% mean absolute success-rate gain.
  • Online autonomous learning delivers a further 32.8% gain over matched offline training.
  • Under the same gradient-step budget, the online schedule performs 8x as many policy iterations, with over 2x lower recorded time per gradient step.

Limitations

The abstract does not report the number of tasks, the distribution of failure modes, or how results vary with the quality of the initial offline dataset. Sparse rewards and off-policy interventions also imply a dependence on reward design and intervention quality; the abstract does not describe the safety constraints or intervention-triggering policy used during real-world runs. The training cost and time-to-convergence for long-horizon tasks are likewise unquantified.

Industry Implications

“Keeps improving after deployment” is one of the most critical gaps in making generalist robots practical: long-tail situations in the real world cannot be enumerated in a lab, so the only way forward is to let robots learn from their own deployment experience. VaM’s value is that it makes this path engineering-grade — value caching answers “it cannot remember task progress,” the online loop answers “feedback is too slow,” and concurrent collection and training directly compresses time cost. The 29.4% and 32.8% gains show both changes genuinely work, and the 8x iteration efficiency shows it is viable under constrained compute. For teams that already deploy generalist policies, this is a concrete blueprint for turning the deployment site into the training site.