RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance

Background

General-purpose reward models have become the bottleneck for scaling robot learning: policies are only as good as the reward or value signal that guides them. Yet most value-related capabilities are still learned from supervision tied to task-internal anchors such as human preferences or normalized progress. Preference labels are expensive to collect and do not transfer across embodiments or data sources, which blocks the field from exploiting the large, heterogeneous corpora that already exist. What has been missing is a supervision signal that can be derived automatically at scale and that still measures something meaningful about task success.

Core Innovation

RynnValue replaces preference and progress anchors with temporal distance: the directed cost-to-go from an observation to the language-specified goal. Because temporal-distance labels can be derived directly from timestamps, the method scales to over 7,000 hours and roughly 3 million instruction-conditioned clips without any preference or progress annotation. To keep temporal-value learning reliable at that scale, the authors combine random temporal sampling, temporal-order shuffling, and a value-isolation attention mechanism that suppresses shortcut learning, so predictions stay sensitive to failures and regressions rather than collapsing onto easy statistical cues. The result is a general value model that can be converted into dense rewards through potential-based shaping, giving a practical reward interface for generalist robot policies.

Results

Trained without preference labels, RynnValue attains an average Kendall’s tau_a of 0.675 on the RBM-EVAL-OOD benchmark, surpassing the fully preference-supervised state of the art (0.655) and more than doubling a progress-only counterpart (0.292). It generalizes zero-shot to unseen tasks, embodiments, and viewpoints. When its outputs are converted into dense rewards via potential-based shaping, real-world policy success improves from 52.5% to 72.5% online and from 63.8% to 82.5% offline, demonstrating that the learned value signal transfers into actual manipulation performance.

Limitations

The paper demonstrates strong benchmark and real-world results, but the evaluation focuses on manipulation tasks and a single model family, so generalization to other robot morphologies and long-horizon tasks remains to be shown. The temporal-distance objective presupposes that a language-specified goal and a meaningful notion of progress exist, which may be less natural for open-ended or exploration-driven tasks. As an open-source release, independent reproduction and stress-testing of the value-isolation attention design will determine how robust the gains are outside the authors’ data distribution.

Industry Implications

Reward and value models are increasingly the practical bottleneck in commercial robot learning pipelines, where preference annotation is costly and task-specific reward engineering does not scale. An open-source value foundation model trained on timestamp-derived supervision offers a route to dense, generalizable reward signals for a fraction of the labeling cost, which matters directly for companies trying to move from demonstration-collection to autonomous policy improvement. The demonstrated zero-shot transfer across tasks, embodiments, and viewpoints, plus the real-world success gains, make RynnValue a candidate building block for generalist manipulation stacks rather than a single-task reward head.