A matter of trust: The challenge facing automated-driving tech

A comfortable nap portended an unsettling problem.

In the early days of Google's self-driving car project, company employees — you might call them beta testers — were allowed to take home vehicles equipped with autonomous technology.

The project's leaders grew alarmed when they saw just how quickly their cohorts over-trusted the embryonic technology — choosing to relax and fixating on their phones instead of watching the road ahead.

"What we found was pretty scary," John Krafcik, former CEO of Waymo, the project's commercial descendant, would say, recounting the tests in 2017. "They have lost contextual awareness."

Infamously, one tester snoozed behind the wheel. It was a decisive moment, so unnerving to Google officials that they came to abandon the pursuit of automated driving technology that involves the commingling of human and machine in favor of an approach that eliminates human involvement.

Others, notably Tesla, remain bullish on human-machine coexistence, but the concerns over introducing automation into vehicles while still requiring humans to maintain responsibility for operations have hardly dissipated.

Tesla launched the hardware for its Autopilot driver-assist system in 2014 and enabled the feature via over-the-air updates in 2015. This year, the automaker released its "Full Self-Driving" corollary in the same over-the-air manner.

In late October, the company surpassed a $1 trillion valuation, in no small part because of the features, both of which generate considerable enthusiasm among the company's fervent car owners. Tesla's financial success leaves industry experts, at least in some corners, wondering whether Waymo made the wrong decision.

From a safety perspective, however, the concerns continue to mount.

The National Transportation Safety Board has investigated multiple fatal crashes involving Autopilot and coined the term "automation complacency" to describe the manner in which human drivers place too much trust in systems that are assistive and not autonomous.

Federal investigators from NHTSA have spent much of 2021 conducting further probes into Autopilot specifically and advanced driver-assist systems overall. In April, the agency's special crash investigation team began examining 39 incidents over a five-year span that resulted in 12 fatalities. Thirty-three of those incidents involved Tesla vehicles.

More recently, NHTSA began a narrower investigation into a dozen incidents in which Tesla vehicles with Autopilot activated struck parked first-responder vehicles. In September, Tesla pushed an over-the-air update to its Autopilot software — and now regulators want to know why the automaker went ahead with the update without issuing a recall.

The fact that regulators are just beginning to consider the safety implications of software-defined automobiles is problematic in and of itself, says Andrew Maynard, a professor in the School for the Future of Innovation in Society at Arizona State University.

"If a manufacturer said, 'We have come out with a new type of brake, and we are not quite sure whether it's going to work, but we'll put it in some cars and see what happens,' nobody would consider that acceptable," he said. "From a regulatory perspective, it would be utterly unacceptable. So why is it acceptable to do that with software but not hardware?"

In the context of driver-assist systems, it's one of several challenges with no clear answers at a time when experts believe driver-assist features — and not self-driving systems — will be the dominant technology for at least the next decade, at least in vehicles bought and maintained by traditional car owners.

Other challenges are just as fundamental. Humans struggle with monitoring automated systems that work well most of the time — a lesson not just learned by Waymo a decade ago, but one understood across industries from aviation to nuclear energy.

A study issued in September by Massachusetts Institute of Technology researchers underscored that point.

They found that when Autopilot is enabled, Tesla drivers' eyes stray from the road more frequently and for longer periods compared with drivers operating vehicles under manual control. Further study of other automakers' driver-assist systems is underway.

If the systems invite inattention, are they considered safe? Some automakers, including General Motors, are pursuing and promoting driver-assist technology that permits drivers to remove their hands from the steering wheel.

"The question for ADAS safety is, are you building a system that a nonsuperhuman driver can operate net safely?" said Phil Koopman, a professor at Carnegie Mellon University who specializes in embedded safety systems.

"We tell them not to fall asleep, but we know better than that. You can't ignore predictable human response."

As driver-assist systems proliferate across the auto industry, understanding the state of the human driver and their readiness to respond will be more important than ever.

Following multiple investigations into fatal Autopilot-related crashes, the NTSB found that Tesla's capabilities of monitoring driver behavior were insufficient.

Among its recommendations, the board said automakers, standards bodies and NHTSA should all develop better applications and rules for monitoring the state of human drivers.

In April, the Alliance for Automotive Innovation, an industry trade group that notably does not count Tesla among its members, outlined a set of safety principles to address the growing call for driver-monitoring systems.

But those voluntary principles don't wrangle with the thorny problems involved in putting driver-assist systems on the road. They say merely that foreseeable misuse of systems should be "evaluated," for example.

Where the NTSB reports said clearly that Tesla's method of monitoring driver engagement by measuring steering wheel torque was insufficient, the alliance merely says that camera-based cabin systems — preferred by many driver-assist purveyors — should be "considered."

The middling nature of the principles is a smaller problem, perhaps, compared with the embryonic state of driver-monitoring technology. Basic inward-facing camera systems may ensure a driver's head is directionally facing the road ahead. Better systems may track a driver's eye gaze, even through sunglasses.

What's most effective, however, may not necessarily be a vehicular schoolmarm that chirps at a human driver when their attention has already strayed or they've already made an error that needs correction.

"Prediction is the key for prevention," said Erez Aluf, co-founder and active chairman of Adam CogTec, an Israeli startup working on technology that can measure a driver's cognitive capabilities and contribute to enhanced human performance.

In that sense, it's more than driver-monitoring technology. It's driver-management technology, attempting to foster a cooperative relationship between human and machine. "It's not one or one," Aluf said. "It's one plus one."

When a car understands not just whether a driver is staring at the road ahead, but details on how engaged a driver is with the task at hand, it can make better decisions on when to hand over or retain control.

Trust lies at the heart of that delicate collaboration. Frank Flemisch, professor of human systems integration at Germany's RWTH Aachen University, likens it to the intuitive balance a horseback rider learns over time with their animal.

But if instilled trust is required between machine and human driver, it's also required — and currently lacking — on a broader level among drivers; automakers hyping the promise of automated technology; independent third parties, such as academics and insurance companies; and regulators.

Creating that trust is not only about brokering responsibility in the event of a crash. It's about ensuring every stakeholder is developing and deploying technology in a transparent and responsible manner.

"In a better society, everybody involved in that chain, whether it's the CEO of Tesla, the engineers making the product or other people making decisions, they'd have a mentality of socially responsible innovation," said Maynard, the Arizona State professor. "In that model, they'd constantly ask themselves, 'Is this the appropriate thing to do?' If they did that, I think things would be different."