QF3: Making Flow Policies Trainable with Off-Policy Reinforcement Learning

Flow matching taught robots to imitate. QF3 is a reminder that the harder half of the problem — improving those policies through experience, without burning an on-policy budget — is where the real engineering wins have been hiding.

Background

Flow matching has become a standard policy class for learning robot behaviors from demonstrations, and it now underpins much of modern manipulation and locomotion learning. But a pretrained flow policy is only a starting point: reinforcement learning is still critical, both to refine a policy learned from demonstrations and to learn one from scratch through interaction. The catch is that the RL recipes built around flow policies have largely been on-policy, so every gradient step consumes fresh interaction with the environment. That is expensive in simulation and prohibitive on real hardware, which is exactly why an off-policy recipe — one that can reuse past experience — has been the obvious thing to want.

Core Innovation

QF3 (Fast Flow RL with Filtered Q-Gradients) is an online, off-policy algorithm that trains a flow policy with two signals at once. The first is flow matching, which keeps the policy anchored to the behavioral target. The second is the critic’s action gradient, and the crucial detail is how it is obtained: rather than differentiating through the full multi-step sampling process, QF3 backpropagates the critic gradient through a one-step prediction of the flow’s output. This makes the gradient cheap to compute, but it also makes it unreliable in exactly the places where the critic and the one-step prediction disagree with reality. The filtering step addresses this: the critic gradient is applied only to action dimensions that stay near the replay action, so updates are confined to regions where the two sources of information can be trusted together. That single design choice is what turns “off-policy flow RL” from a fragile idea into something that trains.

Results

• QF3 is, to the authors’ knowledge, the first off-policy flow RL method to train humanoid locomotion policies from scratch and transfer them zero-shot to hardware.
• Paired with a high-throughput off-policy training recipe, it trains humanoid locomotion and motion-tracking policies with a 10x wall-clock speedup over FPO++, a recent on-policy flow RL method.
• The same algorithm fine-tunes pretrained flow-based manipulation policies on the ABC-Sim and Robomimic benchmarks, showing it can both learn policies from scratch and refine those obtained from demonstrations.

Limitations

The abstract leads with speed and breadth but is thin on the numbers that would settle the comparison. No task success rates or hardware failure counts are given, so the zero-shot transfer claim rests on a single assertion rather than a distribution of rollouts. How the near-replay filtering threshold is chosen, and how sensitive training is to it, is not analyzed — a real concern given that this threshold is the paper’s central mechanism. The manipulation results cover two simulation benchmarks and start from an already-pretrained policy, so the breadth of the fine-tuning claim is unclear. Because the 10x speedup is attributed to the algorithm “paired with” a high-throughput recipe, it is also hard to separate the algorithmic contribution from the training-infrastructure contribution.

Industry Implications

Flow-based policies are moving from research into commercial robot stacks, and the traditional way to improve them — on-policy RL — is precisely the part that is hardest to run on real hardware. An off-policy method that is an order of magnitude faster in wall-clock time and that demonstrably transfers locomotion policies to hardware changes the economics: online improvement of a deployed policy becomes affordable, and learning from scratch no longer requires an on-policy budget. The filtered-gradient idea is also portable — it is a general recipe for trusting a critic only where a cheap one-step proxy agrees with it — which is the kind of reusable trick that tends to outlive the specific benchmark it was introduced on.