Navigating the morass of self-driving car data

Millions of miles are driven, billions of dollars are spent and thousands of hours are devoted to collecting data on which self-driving systems can be trained and vetted.

Much of it is wasted. Enormous efforts are expended to gather and store petabytes of data, but many industry experts lament that this information is banished to hard-to-search repositories where it's often never unearthed.

Elektrobit, a German supplier of embedded and connected software, has sought to provide an antidote. The company, a subsidiary of Continental, has developed a platform intended to ensure that developers of driver-assist and self-driving systems can better organize and access these data troves.

The company's EB Assist Test Lab, which was showcased Monday in Los Angeles, could help system engineers shrink development cycles and reduce the number of miles companies drive to gather data and validate their systems.

"As an industry, we've done this the wrong way twice so far," said Stephanie Hollyer, head of validation software engineering at Elektrobit. "The first way is they say, 'I need to go find a very particular scenario on the road,' and that can take weeks. The second is they have storage rooms full of data, and they don't know how to search it."

The company touts the EB Assist Test Lab as a first-of-its-kind tool, and it provides some novel advances. It encompasses the entire testing cycle, from data gathering during on-road testing or in simulation, to annotation and processing through data analysis.

Driving-scene data can be generated in real and simulated test drives. The tool allows for the import of data and scenarios from third parties. It can collect not only raw sensor data, but any data that travels along the Controller Area Network bus in vehicles.

This data can be unwieldy. For example, camera data alone can be collected at a rate of two gigabytes per second. Size and complexity is only expected to grow for both driver-assist and self-driving functions. A recent report from global consulting firm McKinsey said validation and verification of software will account for 29 percent of a total automotive software market expected to be valued at $24 billion by 2030.

At times, it might have felt like it would take a decade to get all the data from a car and shaped into coherent fashion.

"A year ago, I'd have a petabyte, go get a cup of coffee and wait three weeks," Hollyer said. "Now, you can have everything in a matter of minutes in the cloud, and anyone can search it."

It would be easy to assume, in an age of artificial intelligence, problems with finding stored data would be relics from the past. But in an industry spending billions to put self-driving technology on the road, these data blind spots and workflow complexities remain an everyday hindrance.

Since the Elektrobit technology enables access and offers searchability, companies can better use data they've collected, annotated and stored. That potentially reduces the need for dozens — or in a handful of cases, hundreds — of cars needing to roam roads in hopes of stumbling across interesting or edge-case scenarios.

Instead, development teams can focus their time on finding specific conditions. And they can use the EB Assist Test Lab to benchmark the performance of components and compare the quality of data they get from various sensors in real-world conditions.

"We've worked closely with our customers to identify the pain points in the current test process, and build, from the ground up, a new type of product that addresses every need in the validation and verification workflow," said Martin Schleicher, executive vice president of business management for Elektrobit.