What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policies

· Editorial Team estimated
imitation-learning VLA visual-grounding robustness manipulation

This paper diagnoses why visuomotor imitation policies fail when visually similar distractors appear, framing the problem as conditional visual grounding: the visual target needed for control changes with the manipulation phase and, in complex tasks, with the observed task state. Using Action Chunking with Transformers (ACT), the authors systematically introduce distractors with controlled color and shape similarity and localize failures to picking and placement stages, finding sensitivity depends on both similarity type and stage. Guided by the diagnosis, they combine distractor augmentation, phase-dependent attention regularization, and appearance-based visual prompting, substantially improving robustness in simulation and on a physical UR3e, and show the same failure pattern in a pretrained vision-language-action policy on a state-conditioned instrument-handling task.

Paper · arXiv:2609.05376

A visuomotor imitation policy can ace its training distribution and then fail the moment a visually similar object appears next to the target. This paper argues the failure is best understood as broken conditional visual grounding: the visual target required for successful control is not fixed — it changes with the manipulation phase, and in state-conditioned tasks it changes with what the policy observes. Using Action Chunking with Transformers (ACT), the authors turn this intuition into a controlled diagnosis and then into concrete fixes validated on a physical robot.

Core Innovation

The contribution is a staged diagnosis plus targeted interventions. The authors systematically introduce distractor objects and receptacles with controlled color and shape similarity, then localize failures to specific phases — picking versus placement. This yields a sharper conclusion than “policies are not robust”: distractor sensitivity is specific to both the kind of visual similarity and the manipulation stage, which explains why naive data augmentation alone is often ineffective. Guided by that map, they evaluate three complementary interventions — distractor augmentation, phase-dependent attention regularization, and appearance-based visual prompting — aimed at improving target selection while preserving the spatial information control still needs.

Results

  • Distractor failures in ACT localize to picking and placement, with sensitivity depending on both similarity type (color/shape) and stage.
  • The three interventions substantially improve robustness in simulation and on a physical UR3e.
  • The same failure pattern appears in a pretrained vision-language-action policy on a state-conditioned instrument-handling task, where the observed state of a medical instrument determines the correct destination.
  • Explicitly improving target selection recovers performance across distinct policy-learning regimes, even when the underlying manipulation skill itself is intact.

Limitations

The study centers on ACT-style imitation policies and controlled distractor settings; the intervention mix is evaluated jointly in places, so the individual contribution of each fix is only partially isolated in the abstract. Physical-robot validation, while valuable, covers a bounded set of tasks, and the visual-prompting intervention’s dependence on reliable appearance descriptors in unstructured environments needs further scrutiny.

Industry Implications

Policies that silently grab the wrong object when a look-alike enters the scene are a well-known deployment headache for warehouse, assembly, and surgical-assist automation. This work gives teams a diagnostic vocabulary — “which stage, which similarity” — and an intervention toolbox that does not require new data at scale. For groups shipping visuomotor or VLA policies, the phase-dependent attention-regularization and prompting recipes are directly adoptable, and the instrument-handling result signals that state-conditioned medical and assistive tasks face the same grounding risk.