Autonomous vehicles are cars and trucks that drive themselves, with sensors and software for perception, prediction, planning and control. The field runs from ALVINN, a 1988 neural network that steered a van from camera and laser range-finder imagesSource 1, to Waymo robotaxis that gave more than 500,000 fully autonomous rides a week in April 2026Source 22.

A white Waymo Jaguar I-Pace with a roof-mounted sensor dome and sensor pods on its fenders driving through a San Francisco intersection, the downtown skyline behind it and nobody in the driver’s seat.
A Waymo Jaguar I-Pace driving itself through San Francisco in 2023. Waymo's 2025 study of 56.7 million rider-only miles found statistically significant crash reductions against human benchmarks in every outcome it could test.
Photo: Dllu / Wikimedia Commons. Licence: CC BY 4.0.

ALVINN trained on simulated road imagesSource 1. By 1991 it learned to drive the Navlab van in under five minutes by watching a human driverSource 23. On the 1995 No Hands Across America tour, CMU's RALPH program steered for 2,797 of 2,849 miles, 98.2 percent, while the researchers handled throttle and brakeSource 2. Stanley won the 2005 DARPA Grand Challenge with software built mainly on machine learning and probabilistic reasoningSource 24.

The driving stack

A production autonomous vehicle is a pipeline of perception, prediction, planning and control, each with its own models, metrics and failure modesSource 8 Source 7. UniAD calls this "modular tasks in sequential order"Source 8. Paden's survey adds a vehicle-control layer below motion planningSource 7.

The four stages of the autonomous-driving stack: perception, prediction, planning, and control, with the question each answers, representative methods, and its characteristic failure.
StageQuestion it answersRepresentative methodsCharacteristic failure
PerceptionWhere is everything, and what is it?Lidar, camera, and radar detection and tracking; occupancy grids; the Waymo Open Dataset lineageA missed or misclassified object propagates to every stage below; the Tempe 2018 crash began as a classification flip
PredictionWhat will every moving thing do next?Graph networks over agents and HD map (VectorNet); joint forecasting (UniAD); multi-modal futuresForecasting only the most likely future is unsafe when the rare future is the dangerous one
PlanningWhat should the ego vehicle do about it?Imitative planning with synthesized worst cases (ChauffeurNet); joint training (UniAD); text trajectories (EMMA)Causal confusion and imitation shortcuts: the policy learns what the expert usually did, not why
ControlHow is the plan tracked through the car?Lateral and longitudinal feedback, MPC and the survey taxonomy of Paden et al.Tracking error grows at the limits of available friction and actuator latency

Perception

Perception turns sensor streams into a model of the scene: which objects exist, where they are, how they move and what the road permits. Waymo's encoder fuses camera, lidar and radar inputs over timeSource 25, and Waymo argues that no single sensor can replicate that redundant viewSource 26. BEVFormer, a camera-only bird's-eye-view detector, matched lidar-based baselines on the nuScenes test setSource 27.

Close-up of the roof of a white Waymo Jaguar I-Pace: a domed lidar housing on a raised black mount, ringed by smaller camera and radar windows.
The roof sensor package of a Waymo I-Pace in San Francisco, 2025. Waymo fuses lidar, camera and radar, paying for active range sensing that works in the dark and measures geometry directly.
Photo: 9yz / Wikimedia Commons. Licence: CC BY 4.0.

nuScenes, 1,000 scenes of 20 seconds, was the first dataset with a full vehicle sensor suite: 6 cameras, 5 radars and 1 lidarSource 28. The Waymo Open Dataset added 1,150 scenes of synchronized lidar and camera dataSource 4. By its own metric it was 15 times more diverse than the largest earlier camera-plus-lidar setSource 4. Box detectors struggle with general, out-of-vocabulary objects, so the Occ3D benchmark predicts the occupancy and semantics of the whole sceneSource 29.

Every later stage inherits a perception error. At Tempe in 2018 the system first detected a pedestrian 5.6 seconds before impactSource 14. It classified her in turn as a vehicle, an unknown object and a bicyclist, and never predicted her path correctlySource 14.

Prediction

Prediction estimates what every other agent will do over the next several seconds. The future is multi-modal: a pedestrian may cross or wait, and a predictor that averages the two forecasts a path through neither. The Waymo Open Motion Dataset made interactive forecasting a benchmark: over 100,000 scenes of 20 seconds at 10 Hz, more than 570 hours over 1,750 km of roadsSource 30.

Early predictors drew agent histories and map features into a top-down image for a convolutional network. VectorNet replaced that raster with a hierarchical graph network over vectorized polylinesSource 5. It matched or beat the raster with over 70 percent fewer parameters and an order of magnitude less computationSource 5. MotionLM then cast multi-agent forecasting as language modeling over discrete motion tokensSource 31.

Planning

Planning decides the route, lane, speed profile and trajectory for the next few seconds. CMU's Boss won the 2007 DARPA Urban Challenge with a three-layer planner for mission, behavior and motionSource 32. Paden's survey splits urban planning the same way, with feedback control belowSource 7, the families covered under motion planning.

Front view of Carnegie Mellon's Boss, a black Chevrolet Tahoe numbered 19, with a rack of lidar and camera sensors on its roof and more sensors on its bumper, on display in a darkened hall.
Carnegie Mellon's Boss, winner of the 2007 DARPA Urban Challenge, on display in 2008. Route planning, behaviour decisions and local motion planning, the classical toolbox of its era, remain live production components.
Photo: Steve Jurvetson / Wikimedia Commons. Licence: CC BY 2.0.

On nuPlan, the first closed-loop machine-learning planning benchmarkSource 33, an extremely simple planner built on rule-based priors won the 2023 challengeSource 34. Dauner and colleagues found learned planners limited in complex scenariosSource 34.

ChauffeurNet, Waymo's 2018 imitative planner, found standard behavior cloning on 30 million examples insufficient, even with perception preparing its inputs and a controller executing its outputsSource 6. It perturbed the expert's driving into collisions and off-road excursions and added losses that penalize themSource 6. Manipulation later rediscovered that fix for covariate shift, as behavior cloning foundations explains.

Control

Control turns a planned trajectory into steering, brake and throttle commands, and holds the vehicle on it despite disturbances, tire nonlinearity and actuator delay. Paden's survey covers the feedback path-tracking families: pure pursuit, Lyapunov-based trajectory tracking and model-predictive controlSource 7. A learned planner outputs a trajectory, and a classical controller tracks it.

Modular versus end-to-end

Each interface between stages is a lossy summary, and an upstream error cannot be corrected downstream. A 2024 survey frames the trade: end-to-end systems optimize perception and planning features jointly, while modular designs are easier to interpret and debugSource 9. UniAD kept the task structure but trained it as one network, with stages linked by unified query interfacesSource 8. Waymo's EMMA, built on a multimodal large language model foundation like Gemini, maps raw camera data to driving outputs written as textSource 10.

On NAVSIM, open-loop results on real data did not reflect closed-loop performanceSource 35. Simple methods such as TransFuser matched large architectures such as UniADSource 35. Waymo says its own foundation model has significant benefits over both pure end-to-end and modular approachesSource 25.

Why AV is not solved

Robotaxis now run at scale, and driving is still not solved. Waymo says it gave 15 million rides in 2025 and passed 20 million lifetime ridesSource 36. By April 2026 it operated in 11 major US citiesSource 22.

The levels and the operating domain

SAE J3016 defines six levels of driving automation, 0 through 5. Its 2021 revision named levels 1 and 2 "Driver Support Systems" and clarified the line between levels 3 and 4Source 12. Waymo calls improving a level 2 driver-assist system toward full autonomy "a false summit"Source 26.

Every deployed level 4 system runs inside an operational design domain, a defined geography, road set, speed range and weather envelope. Waymo's crash comparisons align the human benchmark to the same vehicle types, road types and locationsSource 15.

On June 24, 2026, the UNECE World Forum for Harmonization of Vehicle Regulations adopted the first global regulations for fully autonomous driving systemsSource 37. Manufacturers must show that test environments, including virtual testing tools, meet strict credibility criteria, and that the system poses no unreasonable riskSource 37. In the US, DOT and NHTSA announced an AV framework on April 24, 2025. On March 16, 2026, NHTSA proposed amending federal safety standards for automated vehicles without manual controlsSource 38.

In July 2026 NHTSA exempted Zoox's purpose-built robotaxi from parts of eight of those standards for commercial deploymentSource 39. The exemption runs two years, for up to 2,500 vehicles in any 12-month periodSource 39.

The long tail

The technical obstruction is the long tail: scenarios that are individually rare and collectively decisive, such as an officer waving cars through a red light. No driving log holds enough of any one of them to learn from. Waymo's WOD-E2E benchmark collects 4,021 segments, about 12 hours, of scenarios that occur less than 0.03 percent of the timeSource 40.

Generative simulation is the industry's main answer. Waymo's World Model, built on Google DeepMind's Genie 3 and announced in February 2026, generates camera and lidar data controllable by driving actions, scene layout and languageSource 17. Waymo built it to rehearse events "from a tornado to a casual encounter with an elephant"Source 17. The same post reported nearly 200 million fully autonomous miles on real roads and billions in virtual worldsSource 17.

Wayve's GAIA-2 likewise generates multi-camera driving video for common and rare scenariosSource 41. Whether such models cover the tail or only what their training data showed is open, as it is for generative video in manipulation.

The tail still produces recalls. In December 2025 Waymo recalled software on 3,067 vehicles that resumed driving past school buses with red lights flashing or stop arms extendedSource 42. In May 2026 it recalled 3,791 vehicles that could slow but not stop for an untraversable flooded lane on higher-speed roadsSource 43.

The measurement problem

Kalra and Paddock's 2016 RAND study found that test driving alone would need hundreds of millions of miles, sometimes hundreds of billions, to show safetySource 13. Mobileye's Responsibility-Sensitive Safety defines safe longitudinal and lateral distances as a white-box, interpretable modelSource 11. Its authors warned that without such standards the field risked a "winter of autonomous driving"Source 11. Koopman argues that "safe enough" also requires no risk hot spots, such as reckless driving or risk shifted onto vulnerable populations, plus standards conformance and accountabilitySource 16.

US companies must report crashes but not the miles they log, so crash rates are unknown for operators that do not volunteer mileageSource 44. NHTSA has also streamlined its Standing General Order on crash reportingSource 38. California's DMV says its disengagement reports are not designed for comparison across companies, and deployment-permit operations need not report disengagements at allSource 45.

In September 2026, Chris Paxton argued that an automated system should be judged by its potential for catastrophic failure, which is much higher for a truck: it needs far longer to stop, and when one breaks down on a highway the law requires warning flares that a truck with no one aboard cannot set outSource 21. He expected driverless trucks to spread over the following year mainly in China and the US Sun Belt, and put highways where ice forms years awaySource 21.

The deployment record

Waymo's peer-reviewed 2025 study covered 56.7 million rider-only miles through January 2025Source 15. Against human benchmarks on the same roads, it found statistically significant lower rates of any-injury-reported, airbag-deployment and suspected-serious-injury crashesSource 15. Vehicle-to-vehicle intersection crashes with any injury fell 96 percent, with no significant disbenefit in any of its 11 crash-type groupsSource 15.

Waymo's own safety hub reports 271.3 million rider-only miles through June 2026Source 46. Over the same distance it counts 95 percent fewer serious-injury-or-worse crashes and 82 percent fewer injury-causing crashes than human driversSource 46. IIHS found that Waymo's driverless vehicles in San Francisco, Phoenix, Los Angeles and Austin had 68 percent fewer police-reportable crashes per mile than human driversSource 44.

Tesla says it launched unsupervised rides in Miami, Orlando and Tampa in July 2026 and expanded its unsupervised area in AustinSource 47. Baidu reported in August 2026 that Apollo Go had reached 28 cities and more than 240 million fully driverless kilometresSource 48. Aurora began driverless customer deliveries between Dallas and Houston in May 2025 after closing its safety caseSource 49. By its count it drove nearly 440,000 driverless miles through June 2026 with zero collisions attributed to its system, and expected to end the year with 200 driverless trucksSource 50.

On March 18, 2018, an Uber ATG test vehicle in Tempe, Arizona struck and killed a 49-year-old pedestrian crossing N. Mill Avenue at night, pushing a bicycle, where there was no crosswalkSource 14. The system judged a collision imminent 1.2 seconds before impact, and its design precluded emergency braking for crash mitigation alone, relying on the operatorSource 14. The operator was streaming a television show, glancing down until about one second before the crashSource 14. Every layer failed at once, the correlated-safeguard problem the reliability gap article examines.

On October 2, 2023, a driverless Cruise vehicle dragged a pedestrian about 20 feet before stopping. Under a September 2024 consent order, Cruise agreed to a $1.5 million penalty for crash reports that omitted those detailsSource 51. In December 2024 GM stopped funding Cruise's robotaxi work, citing the time and resources needed to scale itSource 52.

The relation to robot learning

The imitation toolkit was built on driving before it reached arms. DAgger, the canonical fix for covariate shift, was demonstrated on steering a car in a 3D racing gameSource 3. It cuts the growth of mistakes over the horizon from quadratic to linearSource 3. On 500,000 hours of driving, Waymo found that forecasting and planning improve as a power law in compute, and closed-loop metrics improve tooSource 53.

RAD trains end-to-end policies by reinforcement learning in photorealistic 3D Gaussian splatting replicas, to address imitation's causal confusion and open-loop gapSource 54. It reports a 3x lower collision rateSource 54.

The convergence is now explicit

EMMA is a vision-language model co-trained as a generalist on planning, detection and road-graph tasks, with its driving outputs written as textSource 10. It is a vision-language-action model whose embodiment is a car. A late-2025 survey splits driving VLAs into end-to-end models, which perceive, reason and plan in one network, and dual-system models, where a slow VLM deliberates over fast, safety-critical executionSource 18.

Waymo's foundation model uses such a "Think Fast and Think Slow" design, a sensor-fusion encoder for rapid reactions beside a driving VLM trained using GeminiSource 25. The split mirrors the end-to-end versus hierarchical divide in manipulation and the knowledge insulation case for a separate fast path. NVIDIA reports that its Alpamayo-R1 driving VLA runs on the vehicle in real time at 99 ms latencySource 55.

Shiva Sreeram, Daniela Rus and colleagues at MIT CSAIL and the University of Haifa (October 2026) asked six open video-language models which way the ego car moves in driving clipsSource 56. On 200 simulated forward and 200 backward clips, every model answered forward every time, and the same biases appeared on 60 real videosSource 56. After LoRA finetuning on 80 percent of those clips, all six scored 100 percent on the held-out 80, and the authors conclude that the motion cues are learnable but not reliably exposed through promptingSource 56.

The Waymo Open Dataset set the pattern of shared perception benchmarksSource 4. Open X-Embodiment pools over a million real robot trajectories across 22 embodiments from 34 labsSource 19. Driving data is counted in hundreds of thousands of hours, such as the 500,000 in Waymo's scaling studySource 53, a contrast that the data bottleneck examines.

What differs in kind

A car steers and accelerates through two channels governed by tire friction and well-studied vehicle models. A manipulator moves dozens of joints that make and break contact. The model-predictive control that tracks a car's trajectorySource 7 has no equally clean purchase on a robot wiping a table, the argument of why RL won locomotion.

Evaluation differs too. Waymo says replaying recorded data in open-loop simulation is insufficientSource 26, and nuPlan moved planning evaluation from open-loop L2 metrics to closed-loop simulationSource 33. Manipulation still reports short-horizon success on small trial counts, the subject of the evaluation crisis.

Driving simulation covers rigid-body motion and appearance with no contact-rich phase, and Waymo says its World Model helps it scale its service safelySource 17. Robot learning attempts the same with Genie-class models whose range of actions is still constrainedSource 20.

What to watch

One open question is whether dual-system driving VLAs settle into the shape of hierarchical manipulation systemsSource 18. Another is whether generative simulators extend from appearance into contact. A third is whether robotics adopts AV's safety-case machinery, from formal models like RSSSource 11 to credibility rules for virtual testingSource 37, before robots deploy at scale. Drones covers flight, where a policy trained in simulation has already beaten human champions.