Physical Words: Giving Latent Robot Reasoning a Vocabulary It Can Reuse

Language reasoning knows what to do but not how it feels. Latent reasoning knows how it feels but cannot name it. Atomic action verbs are a cheap way to get both.

Background

Robot foundation models that reason before acting have converged on two families. The first reasons in language: the model writes or thinks through a plan in words, then acts. This carries task semantics well — “pick up the cup, then place it left of the plate” is unambiguous — but language is a poor channel for physical dynamics. Contact stiffness, slip onset and the difference between a grasp that is about to fail and one that is about to succeed do not survive tokenisation into words.

The second family reasons in latent space, predicting future observations or future states directly. This captures physical evolution far better, but the resulting representations have little internal structure: there is no unit you can point at, no primitive you can reuse, no obvious way to say that this stretch of latent trajectory is the same kind of thing as that one. The two families have been treated as alternatives. PhysWord treats them as two halves of one mechanism.

Core Innovation

The central object is the Physical Word: an atomic action verb used as a shared semantic anchor for a continuous representation of physical dynamics.

Concretely, the model predicts the upcoming atomic action words and simultaneously aligns the contextual hidden states that produced those predictions with latent targets derived from the corresponding future observations. The verb supplies discreteness and composability; the latent target supplies physical grounding. Neither is a supervision afterthought — they are tied together at the same point in the network, which is what prevents the words from degenerating into labels that the policy ignores.

From these primitives the authors build a temporally structured PhysWord space, with visual and optional tactile variants of each action word. The tactile variant matters: a word like “press” or “slide” means something different when contact is sensed directly, and giving each word a modality-specific instantiation keeps the vocabulary from collapsing into a vision-only shorthand.

Finally, the model predicts Physical Words autoregressively across future timesteps. Because the words are ordered in time, this captures transitions between execution stages — the moment a reach becomes a grasp, a grasp becomes a lift — and hands the action generator a structured multimodal context rather than an undifferentiated latent vector.

Results

• 90.3% success on UniVTAC, a visuotactile manipulation benchmark.
• 99.6% success on vision-based LIBERO-Long.
• Outperforms competing methods on real-world tasks using bimanual grippers and dexterous hands.
• Generalises to unseen scenarios in further evaluations, and the structured latent reasoning is reported to enable long-horizon manipulation.

Limitations

The approach depends on having atomic action verbs to predict, which implies either annotated demonstrations or a labelling scheme imposed on existing data; the abstract does not say how the vocabulary was built, how large it is, or how a new task’s verbs get added. A fixed vocabulary is also a coverage assumption — contact-rich dexterous manipulation contains behaviours that may not decompose cleanly into named verbs, and the paper’s real-world evidence is reported as “outperforms” without numbers, so the margin on hardware is unknown. The 99.6% on LIBERO-Long leaves almost no headroom, which suggests the benchmark is near saturation and limits how much the result distinguishes this method. Finally, the visuotactile and vision-only results come from different benchmarks, so the claimed benefit of the tactile variant is not isolated by a controlled comparison in the abstract.

Industry Implications

The vocabulary framing has a practical consequence that raw latent reasoning lacks: inspectability at the level of behaviour. If a policy’s intermediate reasoning is a sequence of action words, an operator or a monitoring system can read the intended stage sequence and detect a mismatch before the arm moves — something a latent vector cannot offer. That is directly useful for the supervision layer that industrial deployments need anyway. The same property helps data operations: demonstrations can be segmented and searched by verb, which makes mixed-quality datasets easier to curate and makes failure analysis tractable. The alignment mechanism is also cheap relative to building a separate world model, since it reuses the policy’s own hidden states. The open question for adoption is vocabulary engineering — who maintains the verb set, and how quickly it can be extended when a new product line introduces motions nobody named.