Show-Harness: Just a VLM Agent Can Play Robots

Foundation VLMs already know a great deal about the world: they describe objects, parse instructions, and reason sensibly about cause and effect in a scene. Turning that knowledge into robot actions, however, has remained the hardest step. The dominant recipe is an end-to-end vision-language-action policy that compresses perception and action into one model, but it demands expensive per-embodiment pretraining or large demonstration collections, and the resulting policy degrades as soon as it drifts outside its training distribution.

Core Innovation

This paper starts from a different hypothesis: what is missing may not be model capacity but the right interface. Show-Harness is an embodied harness that places a compact semantic layer between the VLM and the robot. Instead of continuous joint angles or end-effector poses, the VLM is presented with discrete semantic action units it can reason over much as it reasons over language; embodiment-specific interpreters then deterministically ground those decisions into concrete robot actions. The division of labor is the point — the VLM stays directly responsible for fine-grained physical decisions, while the interpreter only translates semantic choices into an embodiment’s execution.

That interface yields two practical paths. First, it directly unlocks closed-source frontier VLMs for zero-shot robot control, with no access to weights and no robot-specific fine-tuning. Second, it adapts small-scale open-source VLMs for low-cost deployment with only a few GPU-hours of tuning.

The authors also introduce GUMI (GUI Manipulation Interface), which extends the same semantic action space to GUI-based demonstration collection: humans and agents can operate and teach robots across embodiments much as one plays a game, removing the need for specialized teleoperation hardware.

Results

• Show-Harness-equipped VLM agents generalize robustly across tasks, embodiments, and environments.
• They outperform representative agentic and VLA paradigms in the reported comparisons.
• The headline claim is that the right interface can unlock substantial embodied capability from foundation VLMs without additional model capacity or costly embodiment-specific pretraining.

Limitations

The abstract is programmatic rather than quantitative: it reports no per-task success rates, task counts, baseline configurations, or hardware platforms, so the margin over VLA baselines cannot be measured from the summary alone. The granularity and coverage of the semantic action units — together with the reach of the interpreters — bound the ceiling of the whole system; a scenario outside the action vocabulary cannot be expressed no matter how capable the VLM. The abstract also does not discuss inference latency against control frequency, or the failure modes of a play-style interface in safety-critical settings.

Industry Implications

If the claim holds, the value is not a new benchmark number but a change in deployment economics: rather than retraining a large policy for every new embodiment, teams can reuse an already-strong foundation VLM and connect it to the robot through an interface. That lowers adaptation cost and, for the first time, makes closed-source frontier models practical to place in a robotics loop. For groups running multi-embodiment product lines, and for those blocked by expensive teleoperation hardware and slow demonstration collection, the trade of interface design for capability is worth serious evaluation.

---