Splat2Act: Personalizing a Generalist VLA with a Single Phone Scan

· Editorial Team estimated
vla sim-to-real gaussian-splatting manipulation personalization digital-twin

A generalist manipulation policy that works in a lab can score 0% pickup success in a real home or office, and collecting demonstrations in every user's space is prohibitively expensive. Splat2Act takes only a handheld phone scan and images of the user's objects and turns them into a photorealistic digital twin with 3D Gaussian Splatting and collision geometry, automatically finds tabletop and non-tabletop receptacles, builds feasible pickup tasks, and generates successful demonstrations with a privileged simulation oracle. Across 101 scanned campus environments with a median navigable area of 104.87 m^2 it produces 852 receptacles and over 68,000 pickup demonstrations. Two-stage fine-tuning of pi-0.5, first on public object assets and then on the user's own objects at the target scene's receptacles, yields 62.5% aggregate pickup success in the target scene, and the same policy reaches 63.9% on a 12-task subset when deployed on a real Stretch 3, close to its 66.7% simulated success.

Paper · arXiv:2610.13945

A policy that picks up objects flawlessly in a laboratory can fail completely on the first shelf it meets in a real home. The question this paper answers is how to personalize it without sending a robot to every user’s building to collect demonstrations.

Background

Generalist vision-language-action (VLA) policies have become the default route to flexible manipulation: one multimodal model that maps language, images and robot state to actions. But generality learned on broad data does not transfer for free into a specific building. The authors measure the gap directly, and it is stark: a strong generalist such as pi-0.5 achieves 0% pickup success in their captured deployment scenes without adaptation. Closing that gap by collecting real demonstrations in each user’s environment, on each user’s objects, is prohibitively expensive; a fleet of homes and offices cannot be hand-taught one shelf at a time. The paper’s premise is that the expensive part, the human in the loop, can be reduced to a phone scan and a few photographs.

Core Innovation

Splat2Act is an automated real-to-sim-to-real pipeline. From a handheld phone scan it builds a photorealistic digital twin of the whole building using 3D Gaussian Splatting plus collision geometry. Unlike earlier real-to-sim work that reconstructs a single tabletop, it operates at building scale: it automatically discovers tabletop and non-tabletop receptacles, constructs feasible pickup tasks on them, and generates successful demonstrations with a privileged simulation oracle. The only human contribution is capturing the scene and the objects; scale does the rest. Applied to 101 scanned campus environments with a median navigable area of 104.87 m^2, the pipeline yields 852 receptacles and more than 68,000 pickup demonstrations, a corpus no manual collection effort would match. Fine-tuning of pi-0.5 then proceeds in two stages. Stage 1 adapts to hundreds of public object assets, drawn either from the deployment scene alone or from all 101 scenes. Stage 2 personalizes to the user’s own objects on the deployment scene’s receptacles.

Results

  • The strongest personalized policy reaches 62.5% aggregate pickup success in the target scene, against the 0% the unadapted generalist manages there.
  • Deployed on a real Stretch 3, the same policy achieves 63.9% success on a 12-task subset, closely tracking its 66.7% simulated success on those tasks.
  • The pipeline is validated across 101 scanned environments (median 104.87 m^2 navigable area, about 7 receptacles each), producing 852 receptacles and over 68,000 demonstrations.
  • Everything above is learned from simulation-oracle data; no real-robot demonstrations in the target scene are required.

Limitations

The success rate is real but partial: roughly six in ten pickups succeed in the target scene, so the policy is a large step above zero rather than a solved system. The approach still depends on the user performing a scan and photographing objects, and on the quality of the Gaussian-splat digital twin and its collision geometry; a scene that reconstructs poorly may not yield usable tasks or demonstrations. Stage-2 personalization is evaluated per deployment scene, and the paper does not establish how well a personalized policy holds up as objects are added, moved, or replaced after training. The two-stage recipe also inherits whatever biases live in the simulation oracle and the public object assets, since those set the ceiling for the demonstrations.

Industry Implications

The practical message is about the economics of personalization. If a phone scan plus object photos is enough to lift a generalist from 0% to roughly 60% on a user’s own objects, then personalization stops being a per-customer engineering project and becomes a data pipeline that can run at fleet scale, which is exactly what consumer and office robotics deployments need. It also reframes what a deployment team ships: not a single generalist, but a generalist plus an automated scene-capture-and-finetune loop. The residual error rate is the caveat to plan around, and it argues for keeping a fallback such as teleoperation or a human confirm step in early fleets. The building-scale digital twin is itself a reusable asset: capture once, then generate tasks and demonstrations for many future policies and objects.