HumanCLAW: Can Vision-Language Models Act Through a Body?

Background

Evaluating whether a vision-language model (VLM) can act through a physical body is fundamentally challenging because the outcome couples the VLM’s decision with motor control. When a task fails, it is impossible to tell whether the VLM made a bad choice or the motor controller simply failed to execute it — for example, losing balance and falling. This coupling obscures the central question of embodied AI: do current VLMs understand their own physical existence?

Core Innovation

HumanCLAW is an evaluation framework that cleanly decouples action decision-making from low-level motor execution. At every step, a harnessed, off-the-shelf VLM issues an atomic skill command (e.g., “walk forward”, “grasp object”). This command is translated into a sub-second chunk of continuous full-body motion with real physical consequences — gravity, collisions, balance — while execution-side disturbances, motor errors, and balance failures are factored out. What remains measurable is the model’s pure action intelligence: its moment-to-moment choice of what the body should execute next.

Based on this framework, the authors build HumanCLAW-Bench: 1,218 long-horizon, egocentric find-navigate-interact episodes across 41 indoor scenes covering diverse tasks.

Results

Nine state-of-the-art VLMs were tested, and none solves the benchmark. The best model reaches only a 16.8% success rate. Critically, the authors show that recognizing the target object is not the bottleneck — VLMs can identify what they need to interact with. What they lack is embodied self-awareness: they lose track of their own body position, fail to tell whether they have reached the goal, and cannot detect whether they have hit an obstacle. The VLMs behave as if they are disembodied observers issuing commands rather than agents inhabiting a physical form.

Limitations

The benchmark uses simplified atomic skill commands rather than continuous low-level control, which may underestimate the demands of real-time full-body coordination. The 41 indoor scenes, while diverse, remain simulated environments. The framework requires a physics simulation backend, which may introduce a sim-to-real gap in the action execution layer.

Industry Implications

HumanCLAW provides a much-needed diagnostic tool for embodied AI development. For companies building humanoid robots or virtual agents that must navigate the physical world, it pinpoints the specific capability gap — embodied self-awareness — that current VLMs fail at. Improving this deficit will require fundamentally new architectural designs rather than scaling existing VLMs, guiding where research investment should flow. The clear separation of perception-level and action-level failures also makes HumanCLAW valuable as a regression benchmark for embodied AI teams.