DataFarm: An LLM Agent Replaces the Teleoperator at $0.80 an Episode

Robot learning has a cost structure problem that no model architecture fixes. Imitation learning needs demonstrations, demonstrations need a robot and a person, and both are expensive. A single task can absorb weeks of teleoperation before a policy is worth deploying, and every new task restarts the clock. Filtering or curating existing datasets helps at the margin, but it cannot escape the fact that the data was already paid for.

DataFarm asks a different question: can a coding agent take over the teleoperator’s job entirely, working in simulation rather than on hardware?

Core innovation

One calibration episode is the entire human input. The agent receives a data request plus a single calibration episode of the robot in the target scene. From that it reconstructs the task in a simulator: it writes the scene, an expert controller that solves the task, and a success judge that decides whether a rollout counts. This is the key architectural bet. Rather than learning a policy from pixels, the system authorises a program that produces ground-truth actions for free.

Because the expert lives in simulation, labels are exact. Every rollout carries perfect action labels, perfectly synchronised camera views, and a deterministic success signal. No hand-labelling, no inverse dynamics model guessing at actions, no timestamp drift between views. The usual weakness of synthetic data — that it looks nothing like the real sensor stream — is handled separately.

A video model closes the visual gap. Each simulated rollout is re-rendered by a video model conditioned on real images from the calibration episode. Critically, all camera views are generated together, so the resulting multi-view footage stays geometrically consistent rather than being stitched from independently hallucinated frames. The output is “real-ready”: synchronised views with actions attached, suitable for direct policy training and real-robot deployment.

Admission control is explicit. Independent judges certify both the expert controller and every rendered episode before the data enters the training set. This matters more than it sounds. A pipeline that can generate unlimited episodes can also generate unlimited plausible-looking garbage, and without an admission gate the failure mode is silent: the policy trains on confident nonsense.

Results

The headline comparison is on five real-robot tasks. Policies trained only on farm data reached 93 successes out of 100 trials, against 84 for policies trained on the same number of teleoperated episodes. Matching teleoperation with zero robot hours is already a strong result; the robustness number is more interesting. Under an unseen tablecloth and coloured lighting, the farm-trained policies kept 87 successes while the teleoperation-trained policies collapsed to 46.

That gap is the most credible evidence in the paper, because it points at a mechanism rather than a benchmark. Rendering each episode from real calibration images with randomised visual context acts like aggressive augmentation, and the policy learns the task rather than the tablecloth. Teleoperation data, collected in one appearance, does not.

The cost accounting is unusually concrete for this literature: about $21 of API usage per task for adaptation, $5 for a render contract, and roughly $0.80 per additional episode covering simulation, rendering and review. One hour of three-view data lands at about $107 all-in, slightly below a $117 teleoperation quote from China and well below the $290 labour-only figure cited for North America.

Limitations

The obvious one is that this is still simulation. A video model can render a convincing tablecloth, but it cannot invent contact physics that the simulator did not model. Tasks where force, compliance, deformability or slip dominate are exactly where the sim-to-real gap is widest, and the five evaluated tasks are not described as contact-rich.

The judges are a single point of failure. If the admission model is systematically fooled — by a rendering artefact, or by an expert that exploits a simulator bug — the pipeline will happily mass-produce bad data with certified labels. The paper reports no audit of judge false-accept rates.

Costs are quoted as API prices, which move. The $0.80 figure also excludes the engineering labour to build and maintain the pipeline, and it assumes the calibration episode is cheap to obtain, which is true only relative to full teleoperation. Finally, five tasks is a small sample; the robustness result in particular would benefit from more environments and more object categories.

Industry implications

For anyone running a data operation, the interesting shift is where the cost lands. Teleoperation cost scales with episodes and robot hours. This pipeline’s cost scales with task setup plus a small per-episode fee, which means the marginal episode is nearly free once a task is authored. That changes the optimal strategy from “collect carefully” to “author many task variants and generate.”

It also changes who can compete. A lab without a fleet of arms can generate a large labelled dataset if it can specify tasks precisely and calibrate once. The scarce skill becomes task formalisation — writing a scene, an expert and a success predicate — which is much closer to software engineering than to robot operations.

The caution for adopters is not to treat generated data as a drop-in replacement. Use it where visual and geometric variation dominate, keep a real-robot evaluation set that is never touched by the renderer, and audit the judges before trusting the labels.