When Instructions Retrieve Trajectories: Diagnosing and Mitigating Generalization Failures in VLA Models

A VLA policy can clear 90% success on in-distribution tasks and still fall apart on the same sentence in a different place. The question this paper asks is whether the model failed to understand the instruction at all, or whether it treated the instruction as a key for retrieving a familiar trajectory.

Background

Vision-language-action models have become a dominant recipe for robot manipulation: feed language, current observation and robot state into one multimodal backbone and regress actions directly. These models routinely exceed 90% success on in-distribution tasks and survive nuisance changes that do not alter the correct action, such as different lighting or backgrounds. The trouble is that this aggregate view hides a subtler failure. When the environment changes counterfactually — same scene, same instruction, but the correct action is now different — the policy keeps executing the old behavior. The task may appear to complete, but on exactly the occasions that require actually listening to the instruction, the policy has not.

Core Innovation

The authors call this instruction-action binding and profile it with behavioral analysis. Replaying failed rollouts of fine-tuned π0.5 and GR00T-N1.7 policies reveals two recurring patterns: the policy holds on to the source task’s trajectory family, or it switches to another demonstrated task. Crucially, both patterns show that language is not discarded — the model reads the instruction, but treats it as a retrieval cue for a familiar trajectory family and then uses visual feedback to adjust execution. Further readouts and interventions tie this choice to task-conditioned internal states, indicating that the behavior is a locatable mechanism rather than random jitter.

The root cause is traced to the imitation objective itself. If the conditional action support in the demonstrations is too narrow — the same sentence is always followed by roughly the same class of motion — then a genuinely scene-grounded solution and an instruction-keyed retrieval of one familiar trajectory are nearly indistinguishable in training loss, so nothing pushes the model toward the former. Equivariant Counterfactual Training (ECT) starts from precisely this point. At the data level it pairs each instruction with valid demonstrations in which distinguishable scenes require different actions; at the loss level it trains each demonstration together with its counterpart in the same update, separating the grounded and retrieval solutions in the gradient. The intervention lives in the training recipe, not in a larger backbone.

Results

• In a controlled LIBERO-PRO comparison, full ECT raises π0.5’s mean position-swap success from 36% to 59%.
• On CALVIN, the ECT loss alone (counterparts already exist in the original data) improves five-task completion without new demonstrations.
• On a real UR5e under a fixed demonstration budget, full ECT raises unseen-position success from 8% to 88%.
• Both the diagnosis and the fix build on existing π0.5, GR00T-N1.7 and standard imitation pipelines, keeping the change focused and identifiable.

Limitations

The fix depends on being able to construct counterfactual pairs that share an instruction, demand different actions and are visibly distinguishable; such pairs do not arise naturally in most data, so they require extra scene design and collection whose cost grows with task complexity. The controlled quantitative comparison centers on position swaps, and it is not shown systematically whether other counterfactual types — changed goal identity, a substituted tool — benefit similarly. The real-robot result rests on a single UR5e and a fixed demonstration budget; the 8%-to-88% jump is persuasive but establishes efficacy in this setting rather than broad cross-embodiment generality. The retrieval interpretation comes from careful mechanistic analysis, which also suggests internal shortcuts may remain that this particular intervention does not remove.

Industry Implications

The most practical takeaway for practitioners is that passing a nuisance-robustness test is not the same as passing a counterfactual test. Pre-deployment evaluation should disaggregate the two, because a policy that stays stable under nuisance shifts may quietly execute the wrong action precisely when it must follow an instruction — an expensive failure mode in shared spaces or safety-sensitive multi-task cells. A second implication lands on data curation: ECT suggests that rather than piling on volume, teams should add paired, scene-distinguishable counterfactual demonstrations for their critical instructions and let them appear together during training. That offers a data-recipe lever that is cheaper and more directly actionable than reaching for a bigger model.