Learning Surgery From the Surgeon's Own Hands: In-Vivo Laparoscopic Appendectomy Without Robot-Collected Demonstrations

Most minimally invasive surgery is still performed by hand, with laparoscopic instruments held directly by the surgeon. When the case ends, so does almost everything that could be learned from it: the endoscope video is archived, and the way the surgeon’s hands actually moved the instrument is gone.

That asymmetry is the starting point of this paper. A robot learning to operate needs observation-action pairs, and in surgery the natural place those pairs exist is the surgeon’s hands during an ordinary case — not in a teleoperated robot case, where they are easy to record but rare. The paper asks whether demonstrations captured from a surgeon’s own instruments, in a normal operating room, are sufficient to train, select and deploy a bimanual surgical policy on live animals.

Background

Hand-held laparoscopic instruments remain the default tool for most minimally invasive procedures, and the reason is not that robots are unavailable but that the instrumented alternative is expensive and not always better. The consequence for machine learning is severe: the only routinely retained record of a procedure is the endoscopic video. Video shows tissue and it shows instruments, but it does not directly give the instrument’s pose and jaw state, which is what a policy needs to imitate. Recovering that state from video alone is unreliable, and the paper’s premise is that the instrument should carry its own sensors instead of being inferred from outside.

Core Innovation

Three design decisions carry the work:

• A self-contained instrument-state logger. The device mounts on the shaft of a standard laparoscopic instrument and recovers its pose and jaw state from an inertial sensor, a time-of-flight sensor and a Hall sensor, with no external camera or tracker in the room. This is what makes routine surgery recordable: nothing in the existing clinical workflow has to change beyond attaching the logger.
• Latency-aligned observation-action pairs. The pipeline measures the latency of every sensor channel against a robot ground truth and aligns the channels before pairing observations with actions. This is unglamorous but load-bearing — heterogeneous sensors sampled at different rates and latencies will otherwise produce subtly mistimed training data.
• The robot is only a clock and an executor. It is the timing reference for sensor calibration and the thing that performs the operation; it collects no demonstrations. Every training example comes from a human surgeon’s hands.

Policy design is then selected by closed-loop rollouts in a physics simulator that was reconstructed from depth maps of an ex-vivo rabbit appendix, before the policy is retrained on in-vivo data. The backbone is DINOv3, fine-tuned, with a diffusion policy head.

Results

• Demonstrations: 849 in-vivo demonstrations recorded from four live rabbits.
• Deployment: the retrained policy was deployed on four additional live rabbits, with electrosurgery armed.
• Outcome: with the surgeon selecting the surgical phase, the policy completed the appendectomy in three of the four animals.
• Released artifacts: both demonstration corpora are released to support future surgical-robot learning research.

The specificity matters. This is not a phantom study, and it is not a cadaver study: the policy was executed on living animals with an energy device active, and it finished the procedure in most of them.

Limitations

The most obvious limitation is that the abstract reports three of four animals, with no explanation of the fourth — a failure mode on live tissue is exactly the kind of detail a deployment team needs, and it is not characterized here. Second, the surgeon selects the surgical phase; the policy is not autonomous over the whole procedure, and how robust it is to a mis-specified phase boundary is not reported. Third, 849 demonstrations from four animals is a small corpus at a single-site scale, so generalization across surgeons, anatomy, tissue condition, instrument choice and animal-to-human transfer is unestablished. Fourth, the simulator used for design selection is reconstructed from depth maps of one ex-vivo rabbit appendix, which makes it a narrow proxy; conclusions drawn from it should be treated as design guidance rather than validation. Fifth, the calibration path depends on a robot being present as ground truth, which is fine in a robot-executed deployment but does not transfer directly to purely human cases. Finally, the abstract reports no comparison against a teleoperated or alternative-policy baseline, and no quantitative safety characterization of tissue interaction.

Industry Implications

The strategic point is about data, not dexterity. Surgical autonomy is bottlenecked by demonstration data, and demonstration data is bottlenecked by the assumption that you need a robot case to collect it. A clip-on logger that survives the ordinary clinical workflow removes that assumption: any laparoscopic case becomes a candidate training episode, in an operating room that was already going to be used. That reframes the data problem from “buy robot time” to “instrument the existing case load.”

For builders in surgical robotics the sequencing is the interesting part. This group did not start with a policy and search for data; they built the instrument-state recorder, verified its calibration against a robot, used a reconstructed simulator as a cheap design filter, and only then committed to in-vivo training and in-vivo deployment. And publishing both corpora lowers the entry cost for everyone else — which, for a field where datasets have been the moat, is a deliberate choice.

The honest framing is that this is preclinical work: three of four animals, surgeon-supplied phase boundaries, no baseline comparison. But it demonstrates the thing that is hardest to demonstrate in this field — that a policy trained on a surgeon’s own instrument motion can carry a bimanual procedure to completion on a living animal with electrosurgery armed. Whether that survives the transition to human anatomy, multiple surgeons and regulatory scrutiny is the next question, and it is now a question worth asking.