Agile-WAM: A Lightweight Tactile World Action Model for Contact-Rich Control

· Editorial Team estimated
world-model tactile-sensing flow-matching contact-rich-manipulation inference-efficiency manipulation

World Action Models (WAMs) go beyond conventional visuomotor policies by jointly predicting future world states and robot actions, letting the policy learn the physical dynamics that support control. Recent tactile WAMs, however, tend to lean on large-scale pretrained generative backbones to capture contact-rich dynamics, which limits inference efficiency and flexible deployment. Agile-WAM addresses this by encoding visual and tactile observations into a shared latent that serves as the source of a direct vision-tactile-to-action flow-matching process, jointly generating latent representations of action chunks and of future visual and tactile latents. The key observation is that vision and touch evolve at inherently different timescales: adjacent visual frames are highly similar, while tactile signals can change abruptly upon contact. Agile-WAM therefore uses multi-horizon multimodal prediction, supervising the visual latent at a larger temporal offset while predicting the tactile latent for the next frame. Across nine simulated and five real-world contact-rich manipulation tasks it outperforms the strongest baseline in success rate while maintaining low inference latency; in five real-world experiments it delivers a 29.4% relative gain in overall success rate at 11.9 ms inference latency.

Paper · arXiv:2609.20761

A World Action Model (WAM) does more than predict actions: it jointly predicts future world states, so the policy can absorb the physical dynamics that make control work instead of inferring them implicitly from an observation-to-action mapping. For contact-rich manipulation that is an appealing direction, because the moment of contact is precisely where purely visual policies tend to break down.

The trouble is how these models are built. Recent tactile WAMs largely import large-scale pretrained generative backbones to capture contact dynamics. The capability is real; so is the cost in inference efficiency and deployment flexibility. And contact control is exactly the setting where slow does not work — the robot has to react to contact on a millisecond timescale.

Core Innovation

Agile-WAM’s answer is to stop generating future video without giving up explicit modeling of the future.

Visual and tactile observations are encoded into a shared latent, which becomes the source of a direct vision-tactile-to-action flow-matching process. In that single process it jointly generates:

  • a latent representation of the action chunk, and
  • future visual and tactile latents.

The future is still represented explicitly — just in latent space rather than reconstructed as pixels or frames.

What makes the design hold together is a physical observation: vision and touch do not evolve on the same timescale. Adjacent visual frames are highly similar and change slowly; tactile signals can change abruptly the instant contact occurs. Supervising both modalities at one time step necessarily wastes capacity on one and underfits the other.

Hence multi-horizon multimodal prediction:

  • the visual latent is supervised at a larger temporal offset, because slow-changing vision is more informative when looked at further ahead;
  • the tactile latent is predicted only for the next frame, because contact dynamics are fast and must be tracked close to the present.

This writes sensor physics directly into the training objective instead of hoping a bigger model will cover it.

Results

  • Nine simulated and five real-world contact-rich manipulation tasks.
  • Outperforms the strongest baseline in success rate while maintaining low inference latency.
  • Five real-world experiments: 29.4% relative gain in overall success at 11.9 ms inference latency.

That latency figure may be the paper’s most consequential result. It moves world models from offline analysis toward a component that can sit inside a high-frequency control loop — and for tactile feedback, latency is the boundary of feasibility rather than a comfort metric.

Limitations

Only a relative gain (29.4%) is reported, with no absolute success rates, so the baseline level and real usability cannot be judged; the unnamed “strongest baseline” and the size of the margin are likewise unclear. The tactile hardware is unspecified — sensor type, calibration, and the volume of tactile data needed for training — which is exactly what determines whether the approach transfers to another robot hand; whether a different tactile sensor demands a fresh data collection is the most practical question in deployment. The 11.9 ms figure comes without a stated test platform (compute, image preprocessing included or not). The nine simulated and five real tasks are not broken down by contact type, object stiffness, or failure mode.

Industry Implications

Contact-rich manipulation is the part of industrial automation that teams least want to touch and cannot easily avoid: assembly, connector insertion, handling of compliant parts, polishing. All of it depends on force and slip information at the instant of contact. Such tasks have historically been dodged with dedicated fixtures and teach-pendant programming, or underwritten by expensive force-controlled hardware. The engineering signal from Agile-WAM is clear: a tactile world model does not have to cost a large generative model. Representing the future in latent space and supervising each modality at its own true timescale is enough to stay within millisecond-level inference. For teams investing on both the sensing side, tactile arrays, and the algorithm side, this lowers the bar on whether the model can run aboard the robot at all; for humanoid and dexterous-hand platforms, a world model in the ~12 ms range turns fast-reacting contact control from an aspiration into a schedulable engineering target.