TacWAM: Anchor-Guided World Action Model with Mechanics-Aware Tactile Prediction

Background

World Action Models (WAMs) learn to predict future states and generate robot actions in a single model, but existing approaches rely mostly on visual futures. Visual prediction captures scene structure and object motion, yet provides limited supervision for the physical quantities that matter during contact-rich manipulation: force, deformation, shear, and slip. Designing tactile futures that carry meaningful physical information — without letting them become privileged cues the action branch can cheat from — is an open challenge.

Core Innovation

TacWAM addresses this with three coordinated components. A Spatially Aligned Fusion (SAF) tactile encoder maps tactile appearance, dense force fields, and deformation flow into a shared latent prediction space, with bilateral force and torque reconstruction preserving global contact information. A tactile history encoder supplies temporal context so future tactile prediction reflects how force and deformation evolve. Finally, Anchor-Guided Tri-Modal (AGT) attention separates current visual and tactile anchors, future prediction tokens, and action tokens — allowing future tactile states to supervise training without being directly readable by the action branch.

Results

TacWAM was evaluated on four real-world contact-rich manipulation tasks spanning fragile grasping, sustained surface contact, and dynamic in-hand manipulation. It achieves an average success rate of 75.0%, exceeding the strongest evaluated baseline by 37.5 percentage points. Staged ablations show consistent degradation when tactile history is removed and when access to future prediction targets is relaxed, confirming that future tactile supervision improves contact-aware action learning when paired with informative tactile representations and deployment-consistent information constraints.

Limitations

The evaluation covers four tasks on real hardware, but the scope — fragile grasping, surface contact, in-hand manipulation — represents a sample of contact-rich manipulation rather than an exhaustive benchmark. The method requires tactile sensors that provide appearance, force, and deformation signals simultaneously, which limits applicability to platforms without such sensing. The balance between future-tactile supervision strength and action-branch information leakage is delicate and may need retuning per task family.

Industry Implications

Contact-rich tasks — inserting, wiping, in-hand reorientation, fragile-object handling — are where current robotic systems fail most often in production. TacWAM shows that tactile world models can convert cheap-to-collect future tactile signals into substantially better policies without leaking privileged information at deployment. For companies deploying manipulation in unstructured settings, this points toward a practical recipe: richer tactile sensing plus world-model-style training supervision, with consistent performance gains over both standard WAMs and action-only policies.