In an effort to prove that its robotic drivers are safer than humans, Waymo has simulated dozens of fatal accidents that have occurred in Arizona for nearly a decade. The Google spin-off has discovered that replacing one of the vehicles in a two-engine accident with its robot-led minibuses would eliminate almost all deaths, according to data released today.
The results are meant to bolster Waymo’s case that autonomous vehicles work safer than people driving. As millions of people worldwide die in car accidents each year, AV operators are increasingly relying on this safety issue to encourage regulators to pass legislation that allows more autonomous vehicles on the road.
But it was difficult to prove, thanks to the very limited number of autonomous vehicles operating on public roads today. To provide statistical support for his argument, Waymo turned to counter-facts, or ‘what if?’ scenarios, which are intended to show how the robotic vehicles would react in real situations.
Last year, the company published 6.1 million miles of driving data in 2019 and 2020, including 18 accidents and 29 near-collisions. In the incidents where safety operators took control of the vehicle to prevent an accident, Waymo’s engineers simulated what would have happened if the driver had not disconnected the vehicle’s self-driving system to generate a counterfeit. The company also made some of its data available to academic researchers.
This work in counter-facts continues in this most recent release of data. Through a third party, Waymo collected information about every fatal accident that occurred in Chandler, Arizona, a suburban community outside of Phoenix, between 2008 and 2017. Focus only on the accidents that occurred within its operational design domain, or the about 100- Waymo area in which the car makes its cars drive, Waymo identified 72 accidents to simulate to determine how its autonomous system would react in similar situations.
Some of these accidents involved one vehicle, while most involved two. For two-vehicle accidents, Waymo conducted separate experiments in which its autonomous vehicles were imitated in the role of each vehicle – first replacing the vehicle that started the accident and then replacing the vehicle that responded to the other vehicle’s actions. For one-vehicle accidents, Waymo only simulated the single vehicle. This left a total of 91 simulations.
The company has reconstructed these accidents and systematically aligned the vehicle’s lane to ensure that its Waymo vehicles are exposed to a similar situation as in the actual fatal accident. Waymo used the same simulation platform that he uses to train and evaluate his autonomous vehicles on virtual roads in ordinary operations.
The results show that Waymo’s autonomous vehicles would “avoid or mitigate” 88 of the 91 total simulations, said Trent Victor, director of safety research and best practices at Waymo. In addition, Waymo’s vehicles would have reduced the likelihood of serious injuries by 1.3 to 15 times for the accidents that were mitigated, Victor said.
“This means that, even if it would not have avoided the accident completely, it would have taken action to reduce the severity of the impact,” Victor said. “If the severity is reduced, the driver is less likely to die.”
When Waymo replaced the vehicle on the instigator of the accident, it found that it “completely avoided” 82 percent of the simulated accidents – the vast majority of them without having to brake hard or avoid evasive action. In another ten percent as an answer, the urgent maneuvers of the Waymo vehicle helped reduce the severity of the crash. The accidents all took place at an intersection where the other vehicle was turning left or cutting straight across the Waymo vehicle, and had little time to react, the company said.
“We are not saying we will eliminate all deaths, but we are saying that the best way to reduce the chances of serious injury is to take an evasive maneuver if possible,” Victor said. “And in all these accident simulations, the Waymo driver undertook evasive maneuvers.”
There have been three incidents of a person dying after being chased by another vehicle. In simulation, Waymo was unable to avoid these accidents when his vehicle was the one driving backwards. “In the case of rear-end accidents, there is not much the answer can do,” said Matthew Schwall, head of field safety at Waymo. “So the Waymo driver really suffers from the same challenge as people do in those situations, that it’s hard to predict fast enough to be able to make an evasive maneuver.”
Twenty simulated accidents caused a pedestrian or cyclist to be hit by a driver. Waymo’s autonomous vehicles avoided 100 percent of the accidents in simulation, the company said.
There is no standard approach to evaluating AV safety. A recent study by RAND concluded that customers are unlikely to trust the government, although US regulators are happy to let the private sector determine what is safe. In this vacuum, Waymo hopes that policy makers, researchers and even other companies can begin to take on the task of developing a universal framework by disclosing this data.
To be sure, Waymo did not subject his findings to a peer-reviewed analysis for publication in an academic or scientific journal, although it would be open for publication in the future, a spokesman said. The simulations were not performed independently of the company, nor were they reviewed by third parties for review before the company made them public.
The company did share its findings with a select group of academic experts to get feedback. Daniel McGehee, director of the National Advanced Driving Simulator Laboratories at the University of Iowa, said Waymo is pushing safety analysis and transparency ‘to a new level.’
Jonas Bargman, associate professor of vehicle safety at Chalmers University of Technology in Sweden, said Waymo’s use of simulation-based assessment is “the scientific forefront” but only one of the components needed to assess the safety of automotive vehicles.
After reviewing the report, Bargman concluded that the ‘Waymo simulation platform is very sophisticated’, referring to the company’s 3D sensor survey models. He singled out the use of counter-factual axle-axis scenarios as ‘extremely relevant to the introduction of Waymo self-driving vehicles to the public’.