2017 NHTSA Report that Tesla Autopilot Cut Crashes 40% Called "Bogus"

Back in 2017, the NHTSA released a report on the safety of Tesla's Autopilot system after the fatal crash of a Tesla owner in 2016. That report claimed that the use of Autopilot, or more precisely the lane-keeping function called Autosteer, reduced crash rates by 40%. 

In that original crash, the owner repeatedly ignored warnings to resume manual control of the vehicle.  Critics questioned whether Autopilot was encouraging drivers to pay less attention to the road.  The NHTSA report appeared to put those concerns to rest.

Later, when a second driver died in an Autopilot related accident, Tesla CEO Elon Musk pointed to the NHTSA study and the 40% increase in safety claim. Now, 2 years after the original report. According to a report by Arstechnica, a third party has analyzed the data and found the 40% claim to be bogus.

Originally the NHTSA data on Autopilot crashes was not publically available when Quality Control Systems, a research and consulting firm, requested it under a Freedom of Information Act request. The NHTSA claimed the data from Tesla was confidential and would cause the company harm if released.  QCS sued the NHTSA and in September of last year, a federal judge granted the FOI request.

What QCS found was that missing data and poor math caused the NHTSA report to be grossly inaccurate.  The period in question covered vehicle both before and after Autopilot was installed, however, a significant number of the vehicles in the data set provided by Telsa have large gaps between the last recorded mileage before Autopilot was installed and the first recorded mileage after installation.  The result is a gray area where it is unknown if Autopilot was active or not.  In spite of this deficiency, the NHTSA used the data anyway.

In the data provided only 5,714 vehicles have no gap between the pre and post Autopilot mileage readings.  When QCS ran calculations again, they found that crashes per mile actually increased 59% after Autopilot was installed.

Does that mean that a Tesla using Autopilot makes a crash 59% more likely?  The answer to that is no for a number of reasons.  First is that the sample size QCS had to work with is a very small percentage of Tesla’s total sales.  Secondly, the data is only representative of vehicles with version 1 of Tesla’s Autopilot, a version that Tesla hasn’t sold since 2016.

Tesla stopped quoting the NHTSA report around May of 2018, possibly realizing something was fishy with the data. They have since taken to their own report stating that cars with Autopilot engaged have fewer accidents per mile than cars without it engaged.  This has some statistical fishiness to it as well.  Autopilot is only meant to be engaged on the highway and due to the higher rate of speed all vehicles have a lower rate of accidents per mile.

We may just have to wait until more data is available to find out if Tesla Autopilot and systems similar to it make crashed that much less likely.