AeroAct: Action-Centered World-Action Models for Language-Conditioned Quadrotor Flight

Language-conditioned quadrotor flight requires grounding semantic goals, anticipating the visual consequences of self-motion, and outputting smooth, dynamically feasible control references—all under rapidly changing first-person views. Existing approaches either use discrete action spaces, high-level waypoints, or instantaneous velocity commands, none of which provide rich supervision about how flight actions will alter future observations. AeroAct introduces a world-action model (WAM) paradigm that jointly predicts both actions and their visual consequences, and is the first WAM demonstrated on a physical aerial platform.

Core Innovation

AeroAct adapts a pretrained video diffusion Transformer to predict local trajectory-action chunks from egocentric visual history, proprioceptive state, and language instructions. During training, future first-person frames serve as dense consequence supervision—the model learns not just what to do, but what the world will look like as a result. At deployment, actions are decoded directly without generating intermediate video frames, keeping inference lightweight.

To generate aligned training data spanning egocentric vision, state estimates, language commands, and dynamically feasible trajectories, the authors built a comprehensive pipeline combining:

• DiffAero for trajectory generation
• Isaac Lab for physics-validated simulation
• 3D Gaussian Splatting renderers for high-quality novel-view synthesis

A low-cost handheld collection device couples a commodity camera with motion estimates to recreate flight-like egocentric trajectories for real-world data acquisition. A self-guidance procedure improves temporal consistency across overlapping trajectory chunks during rollouts.

Results

Closed-loop simulation and real-world experiments demonstrate:

• Improved target tracking when temporal visual context is provided
• Enhanced object-search performance compared to baselines without visual history
• Successful physical execution of WAM-based policies on a real quadrotor platform
• End-to-end language-conditioned navigation with smooth, dynamically feasible trajectories

Limitations

The paper does not report quantitative success rates or ablation statistics in the abstract; full experimental details should be examined in the manuscript. The handheld collection device, while low-cost, may introduce data distribution shifts compared to actual flight. The dependence on a curated pipeline (DiffAero + Isaac Lab + Gaussian Splatting) for training data generation represents a significant engineering investment that may limit reproducibility.

Industry Implications

This work opens the door to language-guided autonomous quadrotors that understand the causal relationship between their actions and visual outcomes. Potential applications include search-and-rescue (natural language tasking for drone teams), infrastructure inspection (“fly around the north face of the tower”), and aerial cinematography. The WAM paradigm, validated here for the first time in flight, is broadly transferable to other aerial platforms and could fundamentally change how autonomous drones are programmed—from waypoint scripting to natural language interaction.