Pictura: Perspective-View Self-Play at Scale for Driving

Background

Autonomous driving policies trained via self-play in simulation have achieved impressive results, but they typically rely on privileged vectorized observations — exact poses and velocities of all agents including occluded ones. This introduces a representation gap with deployed agents that must drive from egocentric camera views alone. The standard fix of distilling a privileged teacher policy into a camera-student leaves the student imitating decisions its own partial view cannot justify, creating a fundamental ceiling on performance.

Core Innovation

Pictura introduces a GPU-accelerated multi-agent driving simulator that renders each agent’s egocentric perspective view at every simulation step, sustaining up to 500K agent-steps per second (2M images/s) on a single H100 GPU. This mitigates the representation gap at its source by making the training observation modality identical to the deployment modality.

Using Pictura, the authors train Alberti — the first large-scale driving self-play policy trained directly from perspective images, entirely without privileged observations. Training spans 50 billion agent steps, equivalent to approximately 35 million kilometers of driving, using plain PPO with no auxiliary supervision or privileged distillation.

Results

Alberti approaches the driving performance of its privileged vectorized counterpart across standard benchmarks. More strikingly, when transferred zero-shot to Waymo Open Motion Dataset layouts re-rendered in Pictura’s simulator, the perspective-view policy outperforms privileged vectorized agents, demonstrating that direct perspective training produces more generalizable driving behaviors.

Limitations

The work currently operates within a simulated environment; the policy has not been deployed on real vehicles. Sim-to-real transfer of the perspective-view policy remains an open challenge, though the absence of privileged observations and use of realistic rendering should reduce the gap compared to prior work.

Industry Implications

For autonomous driving teams, Pictura demonstrates that privileged observations can be eliminated entirely from the training loop — a result with substantial practical implications. If perspective-view self-play can scale to real-world deployment, it removes the entire perception stack as a training dependency and allows driving policies to be learned end-to-end from simulation. The 2M images/s throughput on a single H100 also makes large-scale camera-based training economically viable.