StenoVLA-3D: A 3D-Aware VLA for Navigation Through Gastrointestinal Stenoses

Autonomous endoscopic navigation is not a factory arm with a different gripper. The observations are texture-poor monocular images, traversability through a stenosis has to be judged from geometry rather than appearance, control must err on the side of caution, and evidence of a lesion has to survive the moment it leaves the field of view. A model that does all of this needs geometric grounding, safe control, and episode-level memory at once.

Background

Current VLAs are strong at semantics and instruction following but lean on visual appearance and short-term context, which limits them in two directions: they lack 3D geometric grounding — a real problem in texture-poor endoscopic imagery — and they lack the ability to report on a procedure once it is over, which is exactly what clinical documentation demands. The paper splits the problem into three parts: predict actions from impoverished observations, make safe control decisions, and retain evidence after the lesion leaves the frame.

Core Innovation

• Geometry-gated point-map fusion. Point-maps are injected into the Cosmos-Reason 2 backbone through a learned geometry-gated mechanism, so the policy reasons with 3D structure instead of inferring geometry from RGB appearance alone.
• A temporal state branch. A dedicated branch models traversal progress, so the model knows where it is within the stenosis and can support decisions that need global progress — advancing, or backing out.
• Reasoning and action from one backbone, with specialised heads. The backbone predicts grounded reasoning together with actions, while stenosis shape estimation and the final lesion report are handled by dedicated heads, decoupling control from documentation inside one framework.
• The EndoCausal dataset. An episode-level dataset carrying lesion annotations, action sequences, and temporally grounded reasoning.

Results

• Held-out episodes: 95.2% semantic accuracy and 83.4% action accuracy across 40 recorded test episodes.
• Physical endoscope: 88.9% and 77.8% task success in esophageal and colonic phantoms on a 3-DoF endoscope, 36 trials each, substantially outperforming the evaluated baselines.

Limitations

The abstract does not identify the baselines or the size of the margins, and it gives no evaluation protocol for the generated lesion reports — who grades them, and whether a clinician reviews them, is left open. Physical experiments are phantom-only, 36 trials per phantom, which is nowhere near statistical sufficiency and still a long way from animal or human studies. The scale, annotation provenance, and public availability of EndoCausal are likewise unspecified. Finally, where the point-maps come from — monocular depth estimation, or 3D sensing on the endoscope itself — decides whether this is a new sensor or a new model, and the abstract does not say, which matters directly for deployment cost.

Industry Implications

Surgical robotics is one of the few robotics segments where model performance maps directly onto clinical value, but it is squeezed from both sides by regulation and by data. StenoVLA-3D sends two signals worth noting. First, geometric grounding belongs in the policy backbone rather than in post-processing; in texture-poor endoscopy it is close to mandatory. Second, episode-level evidence retention is written into the model’s outputs, which lines up with the clinical requirement that procedures be traceable. For teams building surgical robot products, the transferable capability is not the 95.2% figure but the output contract of geometry, temporal progress, and structured reporting — which is also, conveniently, the part regulators care about most.