Drones and Aerial Robotics

A quadrotor is among the most agile machines ever built, and it is the robot that learned to control itself first. Racing quadrotors pull forces several times their own weight, exceed 100 km/h, and accelerate at several times gravity in confined spaces Kaufmann 2023↓Champion-level drone racing using deep reinforcement learningElia Kaufmann, Leonard Bauersfeld, Antonio Loquercio et al., Nature 620, 982-987, 2023. The same airframe costs a few hundred dollars, carries its own sensors and compute, and crashes without breaking anything but itself. That combination, extreme dynamics plus cheap iteration, is why autonomous flight went from a controls problem to a learning problem a full decade before manipulation did, and why the results came faster: by 2021 a neural network flew a drone through forests, outperforming classical obstacle-avoidance pipelines Loquercio 2021↓Learning High-Speed Flight in the WildAntonio Loquercio, Elia Kaufmann, René Ranftl et al., Science Robotics 2021, Vol. 6, Issue 59, abg5810, and by 2023 a reinforcement-learning system beat three human world champions at their own sport Kaufmann 2023↓Champion-level drone racing using deep reinforcement learningElia Kaufmann, Leonard Bauersfeld, Antonio Loquercio et al., Nature 620, 982-987, 2023.

This module covers the two halves of aerial robotics that matter to an ML engineer moving into robotics: autonomous flight, where the field's sim-to-real and learning-versus-control arguments were settled first, and swarm roboticsswarm roboticsCoordination of many relatively simple robots through local interaction rather than a central planner, aiming for collective behavior that no individual achieves: coverage, mapping, or search at fleet scale. The classical models treat the group as a dynamical system of pairwise attractions and repulsions (potential fields, in the lineage of Reynolds flocking), which explains collective motion but guarantees neither safety nor speed in clutter. The aerial-swarm literature moved to onboard trajectory optimization: each drone plans in milliseconds from its own sensors while treating neighbors as constraints, so a ten-drone swarm traverses a bamboo forest with no external localization and no global map., where many cheap flying robots coordinate through local interaction. The treatment is honest about scope: these are domains where the core problems are closer to solved than manipulation is, and the reasons trace back to one structural fact about the dynamics.

The hardest constraint on a flying robot is below, and it is not the control law. Drag the perception latency slider up from 70 ms and watch the maximum safe speed fall on all four airframes. No controller recovers what latency has already spent.

Why flight was the first solved testbed

The comparison this wiki keeps returning to is simulability: reinforcement learning wins wherever the transition function is cheap to simulate, and loses where it is not. A quadrotor is the benign case twice over. Its dynamics are differentially flat, meaning the full state follows from a small set of outputs, and a century of aerodynamics gives accurate thrust and drag models. There is no contact: an aerial robot either flies in free space or it has already failed. Sim-to-real for flight therefore needs domain randomization over motor constants and sensor noise, not over friction coefficients and deformation, which is the wall the manipulation literature keeps hitting.

The empirical result that made this concrete came from the racing track. Swift, the University of Zurich system that beat drone-racing champions in 2023, trained its control policy entirely in simulation with model-free reinforcement learning Kaufmann 2023↓Champion-level drone racing using deep reinforcement learningElia Kaufmann, Leonard Bauersfeld, Antonio Loquercio et al., Nature 620, 982-987, 2023. The gap to the physical world was closed not by better physics modeling but by measuring the gap: the team recorded real flight data, fitted residual models of perception noise and unmodeled dynamics, injected them into the simulator, and retrained Kaufmann 2023↓Champion-level drone racing using deep reinforcement learningElia Kaufmann, Leonard Bauersfeld, Antonio Loquercio et al., Nature 620, 982-987, 2023. The same recipe, empirical residual modeling on top of a clean simulator, is what legged locomotion uses and what manipulation still struggles to copy because its residuals are the whole problem.

A year later the same group published the controlled comparison the debate needed. Song and colleagues raced an optimal-control stack against a reinforcement-learning policy on the same platform, method by method, and found the RL controller faster Song 2023↓Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement LearningYunlong Song, Angel Romero, Matthias Mueller et al., Science Robotics 2023, adg1462. Their diagnosis matters more than the result: RL's advantage was not that it optimized the objective better, but that it optimized a better objective. Optimal control decomposes the problem into trajectory planning plus tracking, and the planned trajectory becomes an interface that limits what the controller can express when reality disagrees with the model; RL optimizes the task-level reward directly and absorbs model error through domain randomization Song 2023↓Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement LearningYunlong Song, Angel Romero, Matthias Mueller et al., Science Robotics 2023, adg1462. The flown result was a peak acceleration above 12 times gravity and a peak velocity of 108 km/h Song 2023↓Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement LearningYunlong Song, Angel Romero, Matthias Mueller et al., Science Robotics 2023, adg1462. Read that against this wiki's module on why RL won locomotion but not manipulation: flight is locomotion's argument at higher bandwidth, with the same moral.

The autonomy stack on a flying robot

An autonomous drone runs the same stages an autonomous car does, compressed by three orders of magnitude in size and power. Perception is a camera plus an inertial measurement unit; state estimation is visual-inertial odometryvisual-inertial odometry (VIO)State estimation for a flying robot from camera images fused with inertial measurement unit data: the camera fixes long-term drift, the IMU provides high-rate accelerations between frames, and a filter or optimization over both yields the metric pose and velocity a flight controller needs. It is the standard onboard localization for small drones because it needs no external infrastructure and no GPS, and it is what lets a racing drone estimate its state from its own sensors alone. Swift's perception module pairs a visual-inertial estimator with a convolutional gate detector, fusing both in a Kalman filter to supply the control policy., fusing image features against IMU accelerations to produce the metric pose and velocity the controller needs, with no GPS and no external motion capture Kaufmann 2023↓Champion-level drone racing using deep reinforcement learningElia Kaufmann, Leonard Bauersfeld, Antonio Loquercio et al., Nature 620, 982-987, 2023. Planning is a local trajectory toward the goal; control tracks it at hundreds of hertz. The stages interact exactly as the driving stack does, and one of them, the perception latency, sets a hard ceiling on everything downstream.

Perception latency bounds the whole system

The latency argument is the cleanest result in aerial robotics, and the figure at the top of this module reproduces it. Falanga, Kim, and Scaramuzza asked a direct question: how fast can a drone fly, given the latency of its perception pipeline and the acceleration its airframe can produce Falanga 2019↓How Fast Is Too Fast? The Role of Perception Latency in High-Speed Sense and AvoidDavide Falanga, Suseong Kim, Davide Scaramuzza, IEEE Robotics and Automation Letters 4(2), 1884-1891, 2019? Their answer is a closed-form inequality. A drone flying at speed v perceives an obstacle when it enters the sensing range s, but the perception pipeline delays the information by the latency tau. The time to contact left when control finally acts is s/v minus tau, and the avoidance maneuver at maximum lateral acceleration u takes time proportional to the square root of the obstacle size over u. If the remaining time is shorter than the maneuver, no controller saves the drone: the crash is set by geometry before any control law runs. The maximum speed follows directly, range divided by latency plus maneuver time Falanga 2019↓How Fast Is Too Fast? The Role of Perception Latency in High-Speed Sense and AvoidDavide Falanga, Suseong Kim, Davide Scaramuzza, IEEE Robotics and Automation Letters 4(2), 1884-1891, 2019.

The numbers make the argument sharp. A stereo camera pipeline with a 70 ms latency, the upper bound the study took from a commercial stereo camera's datasheet, supports about 19 m/s on an airframe that can pull 25 m/s^2 laterally, and 25 m/s on one that can pull 50 Falanga 2019↓How Fast Is Too Fast? The Role of Perception Latency in High-Speed Sense and AvoidDavide Falanga, Suseong Kim, Davide Scaramuzza, IEEE Robotics and Automation Letters 4(2), 1884-1891, 2019. The paper's comparison point is the event camera, a sensor that reports per-pixel brightness changes the instant they happen rather than waiting out a frame; at 8 m of sensing range its ceiling runs 7 percent above the stereo one on the study's least agile airframe and 22 percent at 50 m/s^2, 31.03 versus 25.40 m/s in its table, and the paper's own summary of 7 to 12 percent spans the least agile airframe across sensing ranges Falanga 2019↓How Fast Is Too Fast? The Role of Perception Latency in High-Speed Sense and AvoidDavide Falanga, Suseong Kim, Davide Scaramuzza, IEEE Robotics and Automation Letters 4(2), 1884-1891, 2019. The structural lesson generalizes past sensors: latency anywhere in the loop, sensing, inference, or actuation, converts directly into a speed ceiling or a crash margin, and faster platforms tolerate less of it. A vision-language-action model that takes 50 ms to infer one action is slow on any robot. On a fast drone it is also the binding constraint on the whole system.

End-to-end learning skips the interfaces

The modular stack's weakness on a drone is the same one the driving literature found: each interface between stages is a lossy summary, and latency accumulates across the pipeline. The aerial answer arrived in 2021. Loquercio and colleagues trained a convolutional network to map noisy depth images directly to collision-free trajectories, in a receding-horizon loop, trained entirely in simulation by imitating an expert with privileged information Loquercio 2021↓Learning High-Speed Flight in the WildAntonio Loquercio, Elia Kaufmann, René Ranftl et al., Science Robotics 2021, Vol. 6, Issue 59, abg5810. The key transfer trick was realistic sensor noise in training: simulate the noise properties of the real depth sensor and the policy stops overfitting to clean inputs, and it crossed from simulation to real forests, snow, derailed trains, and collapsed buildings with zero additional training on them Loquercio 2021↓Learning High-Speed Flight in the WildAntonio Loquercio, Elia Kaufmann, René Ranftl et al., Science Robotics 2021, Vol. 6, Issue 59, abg5810. Their own framing of the result is the pipeline critique stated plainly: separating sensing, mapping, and planning "can be problematic for high-speed navigation in cluttered environments" because the stages run sequentially, latency compounds, and errors propagate Loquercio 2021↓Learning High-Speed Flight in the WildAntonio Loquercio, Elia Kaufmann, René Ranftl et al., Science Robotics 2021, Vol. 6, Issue 59, abg5810.

Swift sits in the middle of the same spectrum. It is not one end-to-end network: a visual-inertial estimator plus a gate-detection network produce a low-dimensional state estimate, a Kalman filter fuses them, and a small two-layer network trained by reinforcement learning maps that state to collective thrust and body rates Kaufmann 2023↓Champion-level drone racing using deep reinforcement learningElia Kaufmann, Leonard Bauersfeld, Antonio Loquercio et al., Nature 620, 982-987, 2023. The learned part is the control, the classical parts are estimation, and the split follows the latency argument: perception can afford to be a pipeline because its output is a state vector, while control must be learned to exploit the platform's limits. That division, classical estimation under learned control, is the closest thing aerial robotics has to a consensus architecture, and it is the same shape the manipulation field arrived at from the other direction with real-time chunking over a learned policy.

Swarm robotics

The second half of aerial robotics multiplies the robots. A swarm is many cheap platforms coordinating to do what one expensive platform cannot: cover a search area, map a forest, or fly a light show. The field's founding models came from biology, Reynolds' 1987 flocking model being the canonical one Reynolds 1987↓Flocks, Herds and Schools: A Distributed Behavioral ModelCraig W. Reynolds, Computer Graphics (SIGGRAPH '87) 21, pages 25-43 as printed by the fetched Soria reference list; ACM DL machine-walled so canonical pages unconfirmed, 1987, and treat the group as a dynamical system of local attraction and repulsion terms. Those models explain collective motion, and they have been flown: Vásárhelyi and colleagues optimized the interaction terms and flew 30 drones in coherent outdoor flocking Vásárhelyi 2018↓Optimized flocking of autonomous drones in confined environmentsGábor Vásárhelyi, Csaba Virágh, Gergő Somorjai et al., Science Robotics 3(20), eaat3536, 2018. What they do not provide is guarantees, because reactive potential fields have no lookahead, get trapped in local minima, and have no notion of the platform's dynamics.

The table's second and third rows are the two serious answers, and they share one commitment: each drone plans ahead subject to its own dynamics, instead of reacting. Soria, Schiano, and Floreano folded the interaction terms into a nonlinear model-predictive controller, so every drone solves a receding-horizon optimization that contains both obstacle avoidance and neighbor terms; a swarm of five quadrotors navigated a real indoor obstacle field, faster and with fewer collisions than the reactive equivalent, with behavior independent of the environment layout Soria 2021↓Predictive control of aerial swarms in cluttered environmentsEnrica Soria, Fabrizio Schiano, Dario Floreano, Nature Machine Intelligence 3, 545-554, 2021. Zhou and colleagues pushed the same idea onto palm-sized drones: each robot runs a spatial-temporal trajectory optimizer that solves in a few milliseconds from onboard sensing, treats other drones as constraints, and needs no external localization Zhou 2022↓Swarm of micro flying robots in the wildXin Zhou, Xiangyong Wen, Zhepei Wang et al., Science Robotics 7(66), eabm5954, 2022. Ten of them flew autonomously through a dense bamboo forest, the demonstration that made the result famous, with the coordination entirely implicit in the constraints Zhou 2022↓Swarm of micro flying robots in the wildXin Zhou, Xiangyong Wen, Zhepei Wang et al., Science Robotics 7(66), eabm5954, 2022.

The open problem in swarms is not control but the assignment layer above it. Everything cited above coordinates implicitly; none of it decides who goes where. Compared with the manipulation problems this wiki spends most of its length on, that is a tractable frontier, which is the honest summary of the whole domain: aerial robotics solved its hard problems earlier because they were solvable, and what remains is coordination and scale rather than physics.

What the rest of robotics takes from it

Three transfers from flight to the rest of robotics.

First, the latency argument as a design discipline. The perception-latency study is the reference case for reasoning about any robot's speed ceiling from its loop timing Falanga 2019↓How Fast Is Too Fast? The Role of Perception Latency in High-Speed Sense and AvoidDavide Falanga, Suseong Kim, Davide Scaramuzza, IEEE Robotics and Automation Letters 4(2), 1884-1891, 2019, and the same arithmetic binds manipulators: a policy that infers at 20 Hz controls a contact-rich task at effectively 20 Hz regardless of the arm's bandwidth. The real-time execution module covers that budget for manipulation; flight is where the budget was first measured rather than assumed.

Second, the sim-to-real recipe. Empirical residual models fitted from a small amount of real data, injected into an otherwise clean simulator Kaufmann 2023↓Champion-level drone racing using deep reinforcement learningElia Kaufmann, Leonard Bauersfeld, Antonio Loquercio et al., Nature 620, 982-987, 2023, plus privileged-learning imitation with realistic sensor noise Loquercio 2021↓Learning High-Speed Flight in the WildAntonio Loquercio, Elia Kaufmann, René Ranftl et al., Science Robotics 2021, Vol. 6, Issue 59, abg5810, is the toolchain legged locomotion adopted wholesale. Manipulation inherits it piecemeal wherever its contact models are good enough, which is the subject of the parallel-simulation module.

Third, the calibration it offers for claims of progress. When a manipulation system is described as superhuman or champion-level, the honest comparator is Swift, which earned the phrase against named human world champions under race conditions, on the same hardware, with onboard sensing only Kaufmann 2023↓Champion-level drone racing using deep reinforcement learningElia Kaufmann, Leonard Bauersfeld, Antonio Loquercio et al., Nature 620, 982-987, 2023. Flight got there first because its dynamics permit it. The gap between that bar and current manipulation results is the reliability gap this wiki documents elsewhere, and the difference is the structure of the problem, not the effort spent on it. For the version of that bar that has actually been cleared at scale, under regulators rather than race conditions, read the surgical robotics module.