SUN: Persistent Programs For Language-Grounded Control-to-Learning-to-Real Policies

· Editorial Team estimated
model-predictive-control language-conditioned-manipulation reinforcement-learning sim-to-real robot-learning

Kuafu introduces Semantically UNified (SUN) Programs — typed executables that define geometric and contact relations once and compile them into aligned MPC costs, RL rewards, satisfaction predicates, transition guards, and diagnostics. A large vision-language system synthesizes SUN Programs from language and scene semantics, MPC screens feasibility, and the retained semantics train stage-conditioned policies. Across nine tasks Kuafu reaches 82.03% macro-success (vs. 35.67% sparse-reward and 24.75% Stage-BC baselines), generates 10.57x more successful trajectory time per teleoperation hour at 8192-way scale, and its 500-trajectory datasets train DP3 policies to 46.0% simulation and 34.7% physical Franka/Kinova success — all without demonstrations or dense rewards.

Paper · arXiv:2608.31167

Long-horizon manipulation sits on an unresolved tension between two traditions. Model-based control executes precisely specified objectives; learning-based methods amortize that behavior into a reactive policy. Yet existing protocols for bridging them throw away most of the task semantics: rewards end up hand-crafted, and the learned behavior drifts from what the controller actually verified. The result is a pipeline where “what the task means” and “what the policy optimizes” are only loosely related.

Core Innovation

Kuafu’s answer is the Semantically UNified (SUN) Program — a typed executable in which geometric and contact relations are defined exactly once, then compiled into a family of aligned artifacts: Model Predictive Control (MPC) costs, satisfaction predicates, RL rewards, transition guards, and diagnostics. Because every artifact is generated from the same source, the controller, the learner, and the verifier all optimize the same notion of success.

A large vision-language system automatically synthesizes SUN Programs from language and scene semantics. Feasibility is screened by MPC before any training, so policies only learn behaviors that control has already certified. The retained semantics then train stage-conditioned policies, unifying symbolic planning with data-driven execution.

Results

  • Nine tasks, strong margin: 82.03% macro-success, versus 35.67% for sparse-reward and 24.75% for Stage-BC baselines.
  • Data efficiency: at 8192-way parallel scale, Kuafu generates 10.57x the successful trajectory time per hour of human teleoperation.
  • Downstream policies: with 500 trajectories per task, Kuafu’s data trains DP3 policies to 46.0% simulation success (vs. 22.4% for alternatives) and 34.7% on physical Franka and Kinova robots.
  • No demonstrations, no dense rewards: the entire pipeline runs from language plus scene semantics, which is the strongest practical claim in the paper.

Limitations

The evaluation is conducted on nine tasks with a specific set of manipulation skills; how far the synthesis scales to highly deformable objects, dynamic contacts, or long-horizon tasks beyond the tested suite is not yet demonstrated. The reliance on a large vision-language system for synthesis means the upper bound depends on the VLM’s scene understanding. Physical results (34.7%) remain below simulation (46.0%), so the sim-to-real gap is narrowed but not closed.

Industry Implications

For manipulation companies, the practical promise is a step change in data economics: task semantics specified in language are compiled directly into training signals, removing the need for demonstration collection or expert reward engineering. The MPC feasibility screening also provides a safety filter before costly training or deployment. If the approach generalizes to more task families, it points toward a workflow where fleet operators specify tasks in natural language and receive verified, deployable policies — a meaningful step toward practical autonomy for industrial and service manipulation.