Tesla's key statistic in Autopilot defense comes under scrutiny

WASHINGTON -- Tesla Inc.'s ardent defense of its Autopilot system is getting heat from safety advocates who question a key data point the company has been citing to plead its case.

Several times since an Autopilot-linked fatality last month, the electric-car maker has claimed the U.S. government found that an early version of Autopilot reduced crash rates by 40 percent. Several safety experts say Tesla is misstating a conclusion reached by its regulator. Others are calling on Tesla and NHTSA to release the underlying data as automated-driving technology comes under greater scrutiny following some high-profile deaths.

Both the company and NHTSA -- the publisher of the report Tesla has invoked -- have resisted releasing the data, which is the subject of an ongoing public-records lawsuit. Tesla, which can constantly collect information on the acceleration, braking and speed of its customers' vehicles, is standing by its statements.

"If Tesla's going to keep asserting that, and particularly if they're going to keep crediting NHTSA for it, then I think they need to provide the necessary analysis, caveats and qualifications behind that number," said Bryant Walker Smith, a professor at the University of South Carolina's School of Law, who studies driverless-car regulations. "If that is even close to being true, then that is one of the biggest safety advances since the seat belt."

Tesla's claim stems from a January 2017 NHTSA report that closed an investigation into Tesla's driver-assistance system. The fatal crash of a Tesla Model S on Autopilot in May 2016 prompted the probe, which found no defect.

NHTSA looked at incidents that caused airbags to deploy on Autopilot-equipped Model S and Model X vehicles from the 2014-16 model years to compile the report. The agency used those values, plus mileage data that Tesla supplied, to calculate crash rates before and after the cars installed Autosteer, a key component of Autopilot.

NHTSA calculated that 2014-16 Tesla Model S and Model X vehicles were involved in about 0.8 crashes per million miles after Autosteer was installed, compared with 1.3 crashes per million miles before. The agency said the rates reflected all miles driven -- not just those when Autopilot was in use.

"The data show that the Tesla vehicles crash rate dropped by almost 40 percent after Autosteer installation," the agency wrote. Tesla's CEO Elon Musk called this line the "highlight" of the report in a tweet at the time.

Tesla has cited the 40 percent figure in blog posts and emailed statements to reassure the public after the March 23 fatal crash involving a Model X driver in Mountain View, Calif. The latest statement the company sent to reporters this month stated that "NHTSA found that even the early version of Tesla Autopilot resulted in 40 percent fewer crashes and it has improved substantially since then."

The statements are "an overly generous interpretation" of what NHTSA's report actually found, said Nidhi Kalra, a senior information scientist at Rand Corp. who researches autonomous vehicles. "They're taking one piece of data that is missing support and inflating it to something that it doesn't say."

NHTSA didn't provide the mileage figures needed to show whether the crash rates were statistically significant, Kalra said. And since the crash rates were based only on airbag deployments, the figures would exclude minor fender-benders.

"I don't think Tesla's being egregious," Kalra said. While the company's claim is probably legitimate, "this is the time to be precise in our language when we talk about self-driving and safety," she said. "We're in a tenuous place. The public is confused, regulators are under pressure and companies are moving fast."

The Insurance Institute for Highway Safety has observed a less dramatic decrease in crash rates for Autopilot-equipped Model S sedans from the 2012 through 2016 model years. The safety group found the frequency of claims was 13 percent lower for the cars equipped with the driver-assistance system than those without.

The group is unable to attribute that reduction to Autopilot or other technologies on the vehicle due to limited data, said Russ Rader, a spokesman. Airbags don't deploy in a majority of collision claims, meaning IIHS's data captured a larger universe of crashes than NHTSA observed, he said.

Peter Kurdock, the director of regulatory affairs for the Advocates for Highway and Auto Safety, said Tesla's characterization of NHTSA's report is "grossly misleading."

"What NHTSA clearly said was that the safety improvement was not related to actual Autopilot use," he said.

A Tesla spokeswoman referred back to a March 30 company blog post that said said Autopilot-equipped vehicles see one fatality for every 320 million miles of driving, compared with every 86 million miles for vehicles of all manufacturers. She declined to comment further.

Randy Whitfield, president of data analysis firm Quality Control Systems Corp., is hoping the federal court in Washington will bring about more clarity. Last year, his firm sued NHTSA to release the underlying data after a failed public records request. A spokeswoman for the regulator didn't immediately comment.

"It's an extraordinary claim. If that were true, it would be very, very welcome," Whitfield said in an interview. "I just am not sure what the background of that statement is and what data there are to back it up. That's something that should be subject to public scrutiny."

The figures NHTSA used were contained in a broader database Tesla provided as part of the agency's investigation of the 2016 crash, which was high-stakes for an industry that's touted the potential safety benefits of using autonomy to reduce vehicle crashes that are linked to human choices 94 percent of the time.

Tesla asked NHTSA to designate the database as confidential. NHTSA granted Tesla's request and has resisted releasing the data, saying in court filings it agreed with Tesla's argument that it was "likely to cause competitive harm."

"They can't have it both ways, I don't think," Whitfield said of Tesla.