GTA-2: A Multi-VLM Framework for Synthesizing Robot Manipulation Skills via Grounded Task Axes

· Editorial Team estimated
manipulation-skills multi-vlm task-axis zero-shot robot-controllers

GTA-2 is a modular multi-VLM framework that constructs executable, task-bespoke manipulation skills from reusable object-centric task-axis components rather than predicting actions end-to-end or composing fixed primitives. Each skill is represented as semantic subtasks comprising task-relevant keypoints and axes, controller compositions, and scene-dependent parameters. Four specialized VLM agents separately decompose the task, construct an abstract task-axis skill, assign controller parameters, and ground required visual features from RGB-D observations, enabling zero-shot skill generation with no task-specific demonstrations, policy training, or fine-tuning. On 14 real-robot manipulation tasks it achieves 73.9% average zero-shot success, exceeding the strongest baseline by 31.4 percentage points, and targeted human feedback refines an incorrect stage to raise the average to 90.7%.

Paper · arXiv:2609.09808

Robotic manipulation is usually decomposed into behaviors or skills, but those skills are typically predefined for a specific task or made so generic that they cover everything and execute nothing well. The result is a gap between the abstraction a planner reasons about and the geometric, control, and scene-dependent decisions a controller actually needs. End-to-end VLA policies sidestep the gap by predicting actions directly, but they are opaque and data-hungry. GTA-2 takes a third route: build skills that are executable by construction.

Core Innovation

GTA-2 is a modular multi-VLM framework that represents each skill as semantic subtasks made of task-relevant keypoints and axes, controller compositions, and scene-dependent parameters. These are assembled from reusable object-centric task-axis components rather than fixed task-level primitives.

The work is split across four specialized VLM agents, each handling one stage:

  1. Decompose the task.
  2. Construct an abstract task-axis skill.
  3. Assign controller parameters.
  4. Ground the required visual features from RGB-D observations.

This abstraction-to-grounding factorization is what enables zero-shot skill generation — no task-specific robot demonstrations, no policy training, no fine-tuning. It also keeps the intermediate decisions explicit, so a human can supply targeted feedback to fix one incorrect stage while preserving the components that were already correct.

Results

  • Evaluated on 14 real-robot manipulation tasks against a VLA policy (π₀.₅) and two Code-as-Policies baselines using task-axis controllers or conventional robot primitives.
  • 73.9% average zero-shot success rate, exceeding the strongest baseline by 31.4 percentage points.
  • Targeted refinement raises the average success rate to 90.7%.
  • Project page: https://gta2-project.github.io/.

Limitations

The framework depends on four VLM agents and a hand-designed intermediate representation, so success hinges on the reliability of the underlying models and the expressiveness of the task-axis schema. The abstract does not break down success or failure modes per task, nor does it report latency, which matters for interactive use. Because grounding relies on RGB-D observations, performance under heavy occlusion or transparent/specular objects is not characterized.

Industry Implications

For most industrial manipulation, the pain point is not raw policy capacity but the last mile: turning a task description into something that runs correctly on a real arm, and being able to fix it when one stage goes wrong. GTA-2’s explicit, stage-wise skill representation is attractive precisely because it is inspectable and patchable — a human can correct the failing stage instead of retraining a black box. A 31-point zero-shot margin over a strong VLA baseline, combined with a 90.7% refined success rate, suggests that structured, grounded skill synthesis is competitive with end-to-end learning while remaining far easier to diagnose and deploy.