One of the few yardsticks has been the annual disengagement reports manufacturers are required to submit to the California Department of Motor Vehicles. Manufacturers must list and describe circumstances in which their self-driving systems disconnect from driving or proceed into unsafe situations that require a human driver to intervene.
But the disengagement reports are, at best, snapshots more than a comprehensive tool. They do not offer context on the nature of the tests, whether they occur in complex urban environments or rural highways, or weather conditions. Individual operators even say what constitutes a disengagement, and thus a requirement to report, is open to interpretation.
Rand introduced the idea of "roadmanship" as a potential measure for the competence of self-driving systems. Instead of counting crashes or disengagements, a system could measure events correlated with safety outcomes, such as whether a vehicle's actions would have violated a traffic law.
Beyond legality, roadmanship could benchmark how a self-driving system interacts with the traffic around it. It could set safety envelopes, lateral distances and differentiate between which vehicles cause unsafe conditions and which are responding to them. Rand notes the Responsibility Sensitive Safety model proposed by computer-vision supplier Mobileye, which takes human ideas of safe driving and places them within mathematical rules for vehicles to follow, is an example of roadmanship.
"We need indicators before we get to adverse events," Blumenthal said.