For the past seven years, research scientist Bryan Reimer and his colleagues at the Massachusetts Institute of Technology's Age Lab have been collecting data from vehicles equipped with advanced driver-assist technology. They are working with more than 25 companies that are members of MIT's Advanced Vehicle Technology Consortium, an academic and industry group examining system performance and driver behavior with features such as Tesla's Autopilot, Cadillac's Super Cruise and more.
While there's much focus on the absolute number of crashes reported to NHTSA — which is not useless on its own — those numbers lack necessary information about the geographies in which the crashes occurred, the number of vehicle miles traveled with driver-assistance tech activated, or how automakers are collecting and reporting the information. Even within that context, Reimer is circumspect about the value of crash information alone.
"Crashes and incidents are really rare events," he said. "They're outliers. That's why we're not looking at crashes. We're looking at fundamental behaviors that lead to crashes."
In September, an MIT study believed to be the first that used real-world driving data from driver-assist systems yielded nuanced understandings of some of those fundamental behaviors. For example, researchers found that with the Autopilot system enabled, Tesla drivers' eyes strayed from the road more frequently and for longer periods compared with manual driving.
A separate study issued months earlier showed that human drivers using GM's Super Cruise in Cadillac CT6 models were collaborating in unexpected ways with the driver-assist system, taking more active roles in the driving process than researchers anticipated when the system maintains control.
The work serves as a prelude to upcoming studies that will offer more apples-to-apples comparisons on the systems provided by various automakers, which is the sort of insight many initially hoped would be possible with the NHTSA data.
Reimer suggests that if crashes are merely a tip-of-the-iceberg indicator, further work will shed light on how drivers shift their attention between manual and assisted driving, how drivers put their hands on the wheel in systems marketed as hands-free vs. hands-on, how they retake control after periods where an assist system is engaged, and how automation affects speed and more.
"We desperately need to understand the denominators, the frequency of events and the behaviors underlying them to understand the benefits and limitations of automated and assisted driving," said Reimer, who works within MIT's Center for Transportation and Logistics on human-factor issues. "We need to understand which aspects are working well and which ones need refining."
On an individual automaker basis, companies and their engineers are often well versed in the competence and subtleties of their own systems. Where they fall short is understanding how those compare to other developments, and how they practically function in the real world.
Developing and testing something such as a driver-assist system requires more complex testing.