FeelWorld: Visuo-Tactile World Model for Hierarchical Contact Prediction and Planning

· Editorial Team estimated
world model tactile sensing manipulation contact-rich

A hierarchical visuo-tactile world model that jointly predicts future visual latents and three tactile states (contact state, 3D tactile latent encoding force info, slip state). Uses contact-gated asymmetric attention to maintain visual-only prediction before contact and enable joint visuo-tactile prediction during contact. Achieves LPIPS 61% lower than visual baseline after 80-step autoregressive rollout and 81.7% zero-shot planning success.

Paper · arXiv:2607.24267

Background

Humans plan physical interactions by imagining possible outcomes of candidate actions. Existing visual world models primarily capture appearance dynamics while overlooking the tactile states that govern contact-rich interactions, potentially producing futures that appear visually plausible but violate physical dynamics.

Core Innovation

FeelWorld introduces a hierarchical visuo-tactile world model that jointly predicts future visual latents and three tactile states: contact state, a 3D tactile latent encoding force-related information, and slip state. These are predicted by a shared latent dynamics model with explicit supervision.

Key technical contributions include:

  • Contact-gated asymmetric attention: maintains a visual-only prediction pathway during free-space motion, activates joint visuo-tactile prediction only during contact — preventing irrelevant tactile signals from degrading visual prediction
  • Autoregressive rollouts with context noise injection: improves robustness to compounding errors
  • Contact-aware CEM planning: leverages predicted contact and slip states for zero-shot planning

Results

Across chip grasping, fruit grasping, and USB insertion tasks:

  • 10-step LPIPS reduced from 0.084 to 0.058
  • LPIPS 61% lower than visual baseline after 80-step autoregressive rollout
  • Average zero-shot planning success rate of 81.7%

Limitations

Physical availability of tactile sensors remains a deployment barrier. The method relies on GelSight-class vision-based tactile sensors whose robustness and durability in real industrial environments are not yet fully validated. Additionally, interpretability of the tactile latent codes could be further improved.

Industry Implications

FeelWorld provides a new paradigm for incorporating tactile sensing into world models for contact-rich manipulation. In precision assembly, electronics manufacturing, and surgical robotics, tactile feedback is essential for successful operation. This work establishes a complete technical pipeline from tactile data acquisition to world model learning, offering a valuable reference for next-generation dexterous manipulation systems.