TacPAC: Tactile Prediction and Real-Time Action Correction in World-Action Models for Contact-Rich Manipulation
TacPAC turns tactile prediction into real-time action correction inside world-action models. Vision-centric predictions miss the local contact cues that decide contact-rich manipulation, yet naively predicting future tactile observations as extra views recovers only about a third of the achievable gain in the authors' experiments — a timing mismatch, since predictions precede execution while tactile feedback arrives during it. TacPAC caches the predicted contact a planned action chunk was conditioned on, together with the plan's own representation, and a tactile expert reads each newly observed tactile image against that cache to correct only the not-yet-executed actions. Feedback is interpreted against what the plan anticipated rather than in isolation, and a single correction pass costs 20.7x less than regenerating the chunk. On five real-robot tasks spanning precision insertion, fragile-object handling, object reorientation, and long-horizon manipulation, TacPAC leads every task and lifts average success from 22% for its vision-only base model to 64%. Code is open-sourced.
Paper · arXiv:2609.05266In contact-rich manipulation, success is often decided by local contact cues at the fingertips. World-action models guide action generation with predicted future observations, but vision-centric predictions miss exactly those cues. The naive fix — predicting future tactile observations as additional views — recovers only about a third of the achievable gain in this paper’s experiments. TacPAC’s diagnosis is a timing mismatch: prediction happens before execution, while tactile feedback arrives during it, so the two are inherently asynchronous.
Core Innovation
TacPAC reframes tactile prediction as execution-time correction rather than extra prediction. Once the base model has planned an action chunk, TacPAC caches the predicted contact that plan was conditioned on together with the plan’s own representation. A tactile expert then reads each newly observed tactile image against that cache and corrects only the actions not yet executed. The key design choice is that feedback is interpreted relative to what the plan anticipated — distinguishing benign deviations from genuine anomalies — rather than in isolation. One correction is a single pass over the cache, 20.7x cheaper than regenerating the action chunk, which keeps the mechanism viable in the real-time control loop.
Results
- Leads every task across five real-robot settings: precision insertion, fragile-object handling, object reorientation, and long-horizon manipulation.
- Raises average success from 22% (vision-only base model) to 64%.
- Ablations confirm that naively predicting future tactile views recovers only about a third of the achievable gain, supporting the timing-mismatch diagnosis.
- Code is available at github.com/LogosRoboticsGroup/TacPAC.
Limitations
The headline gain is measured against the paper’s own vision-only base model, and tactile hardware is a prerequisite; per-task success breakdowns, robustness to sensor noise and wear, and behavior on very long horizons are not detailed in the abstract and need the full paper and code. Because only not-yet-executed actions are corrected, errors already committed by earlier actions cannot be retraced.
Industry Implications
Precision assembly, electronics insertion, and fragile-goods handling have long suffered from the gap between “visually correct” and “tactilely wrong.” TacPAC offers a pragmatic architecture: instead of making world models regenerate tactile predictions at inference time, use a lightweight tactile expert for cheap execution-time correction — a 20.7x cost advantage that makes it realistic inside a control loop. For teams deploying generalist manipulation policies, the pattern of “plan-conditioned prediction plus execution-time sensor correction” generalizes to other high-frequency channels such as force and audio.