DexTacWAM: A Visuo-Tactile World-Action Model for Dexterous Manipulation

Dexterous manipulation is decided by contact, and contact is exactly what cameras cannot see. Fingertip pressure, the onset of slip, and the transfer of load between fingers are largely invisible to vision, so a policy trained on images alone learns the visible half of the task. World-Action Models (WAMs) looked like the natural answer: couple predictive video world modeling with action generation and let video-pretraining priors carry over into control. But the current WAM generation stays vision-centric, which means the predicted future state — the object of all that prediction — never contains the contact dynamics that the task hinges on.

Core Innovation

DexTacWAM keeps the world-action structure and puts touch inside the predicted state instead of beside it.

• Per-fingertip encoding. Each fingertip is encoded independently rather than collapsed into one tactile vector, so the model retains which finger touched what — the spatial identity of contact survives the encoder.
• Finger- and pose-aware tactile compressor. The per-finger features are aggregated by a learned compressor that is conditioned on finger identity and hand pose, keeping the contact structure intact while reducing dimensionality. The compressor retains 89.4% of pre-fusion contact recall while making training 2.26x faster and inference 1.29x faster.
• Tactile latent injected into a video diffusion world model. The compressed tactile representation enters the world model itself, so joint visuo-tactile world modeling predicts how contact evolves as part of the future state.
• Ablation that isolates the mechanism. Removing tactile world modeling while keeping the same tactile features and action expert cuts the four-task mean from 74.7 to 26.6. The benefit therefore comes from modeling contact evolution as world state, not from conditioning the action expert on touch.

Results

• Six contact-rich dexterous manipulation tasks on a 22-DoF bimanual platform: DexTacWAM achieves the highest score on every task, averaging 70.6 versus 38.0 for the strongest baseline.
• Continual vision-to-touch learning: about four hours of tactile-encoder adaptation with a frozen pretrained vision VAE extends the pretrained video model to touch using roughly 100 demonstrations per task, without tactile midtraining, while retaining visual prediction quality within 0.5 dB of vision-only counterparts.
• Efficiency: the tactile compressor preserves 89.4% of pre-fusion contact recall at 2.26x faster training and 1.29x faster inference.

Limitations

All evidence comes from a single 22-DoF bimanual platform. The abstract reports aggregate task scores but no per-task breakdown, no failure taxonomy, and no comparison across different tactile sensor types or mounting configurations. The data and compute claims — roughly 100 demonstrations per task and four hours of adaptation — are measured on one setup, so how they scale with task diversity, object variety, or sensor hardware is unknown. There is no evidence of generalization to unseen objects or tools, no long-horizon or multi-stage task, and no real-world application beyond the platform itself. The paper also does not report end-to-end closed-loop control frequency: the speedups shown belong to the compressor, while the diffusion world model still incurs generation cost per decision, and that cost is left unquantified.

Industry Implications

Touch is arguably the last high-value sensing modality not yet absorbed into large pretrained robot models, and the usual objection is that building a tactile foundation model is a project in itself. This work argues the opposite route: reuse an existing video prior and extend it to touch with modest data, modest compute, and a frozen vision backbone. For dexterous hands, bimanual assembly, and any task where vision is occluded by the hand, that materially lowers the barrier to contact-aware policies. The engineering questions that remain are mundane but decisive — which tactile sensor and mounting will survive production wear, and whether the diffusion rollout can be kept inside a control period once it sits in the loop.

---