WholeBodyMem: Remembering What the Humanoid Saw Before It Acts

Humanoid loco-manipulation is stubbornly non-Markovian. The robot turns to pick up a tool and the target leaves the frame; it walks toward a table while someone moves the cup it was about to grasp; it bends down and its own arm occludes the camera for a few seconds. By the time the policy must choose an action, the evidence it needs is no longer in the image. Whole-body vision-language-action policies, which see only the current observation, are structurally blind to this.

Core innovation

WholeBodyMem does not try to widen the context window. It decides, first, which history is worth keeping.

SAM3 tracks the task-relevant objects, and object-level events — appearing, being displaced, becoming occluded, reappearing — act as causal cues for selecting key frames under an adaptive budget. Selected frames are not fed in raw. They are encoded into object tokens carrying appearance, geometry and the time of the event, and the action expert reads them through cross-attention alongside the current observation.

The design commitment is explicit: keep the right frames, not more frames. Uniform sampling or a fixed-size memory window fills up with irrelevant frames over time, diluting the few seconds that actually determine success. Event-driven selection is a semantic compression of memory, driven by the task rather than by the clock.

The paper also introduces HuMeGo, a benchmark of six simulated and six real Unitree G1 tasks built specifically around scenarios where cues disappear.

Results

• Simulation: base-policy success 29.1% → 72.6%.
• Real Unitree G1: 36.7% → 88.3%.
• Standard SIMPLE tasks also improve: 70.2% → 86.9%.

The real-robot jump of 51.6 points is the most informative number. It rules out a sim-to-real explanation: the failures were not caused by the reality gap but by the policy operating on the wrong information. The SIMPLE gains matter for a different reason — they show the benefit is not confined to tasks designed around memory, since ordinary manipulation tasks also improve once history is restored.

Limitations

• The abstract reports success rates but no inference latency or memory footprint. Tracking with SAM3, selecting frames, and encoding object tokens is a multi-stage pipeline whose real-time behaviour on robot hardware is the obvious open question.
• The method depends on SAM3 tracking quality. If an object leaves the view entirely or stays occluded for a long stretch, the event stream breaks and frame selection degrades to having nothing to select from; the abstract does not describe a fallback.
• Evaluation covers one platform (Unitree G1) and twelve tasks, so cross-embodiment and cross-scene scaling remain untested.
• How the adaptive memory budget is set, and whether it needs per-task tuning, is not detailed.

Industry implications

The paper names an underrated engineering fact: a large share of humanoid failures in the field are not control-precision problems but policies running on false assumptions about what they can see. Scaling the model does not fix “the cue is no longer in frame.” Memory does.

For teams building whole-body VLAs, the immediately reusable piece is the event-driven frame-selection layer. It sits in front of the policy backbone and can be attached to an existing VLA without retraining the action head. HuMeGo is valuable for the same reason: it converts memory from a qualitative talking point into a measurable axis, so a team can tell which class of disappearing cue is actually breaking its policy.

The question to watch is cost. If SAM3 tracking and event filtering can run at control rate on onboard compute, this kind of memory becomes standard equipment for humanoid VLAs. If it needs off-board compute, it is better understood as a data-collection and evaluation tool than as an online module.