PhysCanvas: A Vision-Tactile World Model for Robot Policy Evaluation and Learning

A world model’s most dangerous failure is getting the contact right on screen. PhysCanvas pulls touch into the representation and then scores the model against real outcomes.

Background

Evaluating and post-training robot policies has always been gated by hardware. Every rollout costs machine time, human supervision and wear, so sample sizes stay small and confidence intervals stay wide. Action-conditioned world models are the obvious escape hatch: if a model can act as an interactive simulator, policies can be rolled out, scored and updated inside it at a fraction of the cost.

The catch is that visually plausible prediction is not physically correct prediction. Contact-rich manipulation is exactly where vision is most ambiguous. Whether a finger is really pressing on an object, whether it is slipping, how large the reaction force is during insertion — in RGB these may amount to a few pixels, yet they decide task success. Prior systems that patch this with post-hoc physics settling often produce scenes that look right and collapse the moment they are manipulated.

Core Innovation

The first decision is to flatten the modalities into one vision space instead of building a bespoke encoder per sensor. Multi-view RGB, tactile images, marker displacement, task state and actions all enter a unified representation sharing one video VAE and one pretrained generative backbone. The practical payoff is that optical touch — contact deformation read out through marker displacement — can reuse visual and temporal priors the model already learned from video, rather than learning tactile dynamics from scratch.

The second decision concerns action conditioning. Forward kinematics plus camera projection generate motion images that align actions with image pixels spatially, while an initial RGB anchor and rolling reference observations constrain sequential generation to prevent long-horizon drift. This addresses the common world-model failure where actions and pixels proceed on separate tracks.

The third decision is the one that shows the most engineering judgement: consistency-aware post-training. Three frozen reward models — a Scene Reward Model for scene preservation, a Modality Reward Model for visual-tactile compatibility, and a State Reward Model for agreement with real task outcomes — supervise the world model. The model is therefore not asked to generate something that looks right, but to predict outcomes that match the hardware.

Results

• Across four real manipulation tasks, adding State Reward Model supervision raises the arithmetic mean of task-wise Pearson correlations between world-model and real-world success rates from 0.602 to 0.885.
• Closed-loop policy rollouts inside the world model are used for evaluation, and its imagined interactions and rewards drive GRPO policy updates.
• After post-training inside the model, pi-0.5’s mean real-world success rate rises from a 59.1% SFT baseline to 70.8%.
• The authors attribute both gains to unified multimodal modelling plus consistency-aware post-training.

Limitations

The evidence covers four real tasks, and the headline number is an arithmetic mean over task-wise correlations, so a single task with poor consistency can hide behind the average; 0.885 is also far from the precision needed to replace hardware evaluation, and policies filtered this way will still fail on the robot. Touch depends on specific hardware (tactile images and marker displacement), so a different gripper or tactile sensor requires re-adaptation, and cross-embodiment transfer is untested. The three reward models are frozen and themselves require real task outcomes to train, which moves part of the hardware cost upstream rather than removing it — the total may not fall so much as shift from per-rollout execution to one-off calibration plus in-model rollouts. Finally, the abstract reports nothing about out-of-distribution objects, unseen contact modes or long-horizon tasks, which is precisely where an evaluator most needs to be trustworthy.

Industry Implications

For manipulation teams, the practical value is that this turns “can a world model serve as an evaluator” into a measurable quantity: not whether generated video looks right, but how well in-model success ranks against real success. The 0.885 figure gives engineering a usable bar — a homegrown model materially below it should not be used to screen policies.

More interesting is the loop where the evaluator is also the trainer. One model both scores policies and supplies imagined interactions and rewards for GRPO updates, and real success improves by 11.7 points as a result. That makes world-model quality compound: a better model screens and trains better policies, and better policies produce better hardware data to calibrate the model. For teams with limited robots but unavoidable contact-rich work, this downgrades hardware from a training necessity to a calibration and acceptance tool, which is the most realistic cost reduction available today. The precondition is willingness to pay the one-off cost of building and calibrating those three reward models, and to accept that the model is only trustworthy inside the contact modes it has seen.