ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation
ViTacWorld is the first visuo-tactile world model framework for contact-rich manipulation. It leverages public real tactile datasets and simulated environments to pretrain visuo-tactile-action trajectories, exploiting the fact that tactile signals exhibit a smaller sim-to-real gap than visual observations. It serves both as a data augmentation engine and a policy evaluation tool, generating physically meaningful rollouts that improve downstream tactile policies.
Paper · arXiv:2607.22530Background
Contact-rich robot manipulation requires physical interaction cues that cameras cannot capture, making tactile sensing essential for robust control. However, scaling visuo-tactile robot learning is difficult because real tactile interaction data is expensive to collect, hardware-dependent, and limited in task and scene diversity. No prior work has demonstrated a world model capable of predicting both visual observations and tactile feedback from actions.
Core Innovation
ViTacWorld introduces an action-conditioned visuo-tactile world model that predicts temporally aligned visual observations and tactile feedback given robot actions. The key insight is that tactile signals are directly grounded in physical contact and exhibit a smaller simulation-to-real gap than purely visual observations, making sim-to-real transfer more feasible for tactile data. The model is first pretrained with large-scale real and simulated visuo-tactile trajectories, then finetuned with real-world policy rollouts. To the best of the authors’ knowledge, this is the first framework that uses a world model for robot visuo-tactile-action trajectory generation and policy evaluation.
Key Results
ViTacWorld serves two roles: synthesizing rollouts to improve downstream tactile policies, and evaluating policies by predicting action-conditioned visuo-tactile outcomes under controlled action sequences. Experiments on contact-rich manipulation tasks show that ViTacWorld generates physically meaningful rollouts, improves policy performance through scalable data augmentation, and enables action-conditioned policy evaluation. The framework leverages both public real tactile datasets and a constructed simulation environment for data scaling.
Limitations
The quality of generated tactile feedback depends on the fidelity of the tactile simulator used during pretraining. The framework has been demonstrated on contact-rich manipulation but its applicability to high-speed dynamic tasks (e.g., impact or sliding) remains unvalidated. Real-world finetuning still requires some policy rollouts, though far fewer than training from scratch.
Industry Implications
Tactile sensing is a critical bottleneck for deploying robots in tasks that require delicate contact — assembly, surgical assistance, food handling, and material sorting. ViTacWorld’s ability to generate synthetic visuo-tactile trajectories at scale could dramatically reduce the data collection burden for tactile robotics. Its dual role as both a data augmentation tool and a policy evaluator makes it a practical asset for any robotics lab working on contact-rich manipulation, potentially accelerating the adoption of tactile sensing in industrial applications.