Distilling RL's Control Knowledge into a Vision Policy

Dexterous manipulation from a handful of visual demonstrations is hard for a reason that is easy to state and hard to fix. A policy has to learn two things at once: what the scene looks like from an image, and how to move a high-degree-of-freedom hand to make and break contact precisely. Imitation learning gives you the second only as far as the demonstrations cover it, and demonstrations are expensive. Reinforcement learning in simulation can discover rich control behaviour, but it does so with privileged information — object poses, contact states, forces — that no camera provides at deployment.

The usual compromise is to train a big vision-language-action model on a large corpus and fine-tune it per task. That works, but it spends data where data is scarcest, and full-parameter fine-tuning tends to overwrite the control competence the model had.

Core innovation

Separate control from perception, and learn them in the order that is cheap. The first stage is privileged pretraining: large-scale RL rollouts in simulation are distilled into a flow-matching action decoder conditioned on proprioception and a latent task context. The decoder never sees an image. It learns a generative map from a compact context to action chunks, which is where the precise, contact-rich behaviour lives.

The second stage is visual adaptation. The decoder is frozen, and the authors learn a visual interface that maps images to the latent context the decoder expects. Crucially, that interface is trained on action supervision alone — no privileged labels, no regression targets on the latent itself. The image encoder is not asked to reproduce an internal state; it is asked to produce whatever context makes the frozen decoder output the right actions.

One decoder, several arms. Because the decoder is conditioned on a latent context rather than on a specific embodiment’s joint layout, the same decoder can be shared across arms by learning coordinated visual contexts for bimanual manipulation. Each camera stream contributes to the context that drives both arms, which is a neat way to get coordination without a separate multi-arm policy architecture.

A design finding worth remembering. The paper compares direct visual-to-latent conditioning against routing visual predictions through the pretrained state encoder. Direct conditioning adapts better and trains more stably. That is a small result with practical consequences: the obvious architectural choice — predict the privileged state, then feed it to the decoder — is not the best one.

Results

Across three simulated tasks, the method outperforms full-parameter fine-tuning of pi0.5 and GR00T N1.7 in the low-data regime. On door lifting and bimanual assembly it surpasses full-data baselines while using 10x and 5x fewer demonstrations respectively. Those are the numbers that matter: not “better than fine-tuning at equal data”, but “better than fine-tuning at full data, with a fraction of it”.

Two pieces of analysis support the claim that the decoder really retains RL-acquired control knowledge rather than merely acting as a good regulariser. The learned representations are robust to unseen mass variation, and contact-force information is more linearly accessible in the decoder’s representations than one would expect from a purely visual policy. Both are consistent with the decoder having internalised a dynamics-aware control prior.

Limitations

The evaluation is three simulated tasks. Dexterous manipulation is exactly the setting where simulation-to-reality gaps are largest — contact friction, tendon compliance, sensor latency and object variability all bite. No real-robot result is reported, so the central claim is demonstrated in the environment where the privileged pretraining happened.

The privileged RL stage is not free. It needs a simulator with reward design and enough parallelism to generate large-scale rollouts, and its quality bounds everything downstream. A frozen decoder also caps how far visual adaptation can go: if the pretraining distribution missed a behaviour, no amount of interface learning recovers it.

Finally, “beats full-data baselines with 5-10x fewer demos” is a strong claim that deserves per-task variance and seed counts. The abstract gives point comparisons without them.

Industry implications

The cost structure this paper implies is attractive. RL in simulation is a one-time capital expense: build the environment, shape the reward, run the rollouts. Adapting to a new camera rig, a new arm, or a new task then becomes a small supervised problem on action labels, which is far cheaper than collecting teleoperated demonstrations at scale.

It also suggests where robot foundation-model competition may shift. If control competence can be distilled once and reused, then the differentiator is not the size of the action head but the quality of the simulator and the reward design that produced the prior — assets that are proprietary, hard to copy, and specific to a robot family.

The practical caveat is calibration. A decoder pretrained with privileged state carries assumptions about proprioception quality and control latency. Deploying it on hardware with different dynamics requires checking that the frozen prior still corresponds to the real arm, which is precisely the sim-to-real work the paper leaves untested.