Kentley-Klay and I stood on a sidewalk near San Francisco's Ferry Building and summoned a car with Zoox's prototype app. A couple minutes later, a matte black Toyota Highlander arrived with a roof rack full of sensors, a trunk full of computers and a passenger cabin full of screens showing what the car can see. After a tap on the app to begin the journey, the SUV merged into traffic, and our 10-minute or so drive commenced. (Zoox's more radical, futuristic vehicle without the steering wheel has yet to hit the San Francisco roads but is being tested at its research facility in Menlo Park, Calif.)
Autonomous cars already do pretty well on the highway, where conditions are more controlled and predictable. The great puzzle in need of solving has been city driving, particularly in a place like San Francisco with its funky roads, streetcars, thick traffic and oblivious pedestrians who value checking their Instagram feeds over maintaining long, productive lives. The company that has received the most press to date for driving in a dense, urban environment has been General Motors' Cruise Automation -- also in San Francisco. Waymo, which is Alphabet Inc.'s self-driving tech arm, has tested its vehicles for years on the sleepier streets of Mountain View, Calif., and more recently in suburban Phoenix, and Uber has done some city driving, too. Since its start, Zoox has been all about making a car suitable for dense, urban streets and seems well on its way based on what I experienced.
During my drive, there were two people sitting up front making sure nothing went really wrong. The driver had her hand covering the wheel at all times, and the passenger had a laptop that revealed in gory detail what the car's computers could see and what the car was expected to do. Kentley-Klay says that human intervention remains necessary on some of the drives, but the humans were never summoned during my test. This was a 100 percent robot affair.
The screens inside the car depict a world rich with information. The computer assigns different colors to different objects: Buildings are gray; cars are purple; stoplights are pink; and pedestrians are orange. The computer gives a unique identifier to every object and provides details on that object's location and speed. Even while classifying objects, the Zoox software can do pretty nuanced things like telling the difference between a person on a motorcycle and another on a bicycle going at similar speeds or when it should give a parked car some extra space because a person is about to open the door.
During the straightforward parts of the drive, the Zoox vehicle behaves much as one guided by a human would. It follows the car in front at a safe distance and observes the actions going on around it. The major thing I noticed during the stop-and-go traffic was that the car seemed, now and again, to stop more abruptly than a decent human driver would.