Next month, California’s Department of Motor Vehicles will release the latest batch of reports from companies testing self-driving vehicles in the state. But the reports offer, at best, a flawed glimpse into the slow and often opaque process of testing autonomous vehicles on public roads, says Kyle Vogt, co-founder and CTO of GM-based Cruise. In a Medium post, Vogt posits that maybe it’s time for a new metric for reporting safety of self-driving cars.

By law, all companies that are actively testing self-driving cars on public roads in California are required to disclose the number of miles driven and the frequency in which human safety drivers were forced to take control of their autonomous vehicles (also known as a “disengagement”).

But Vogt slams the notion that the disengagement reports are useful for tracking technological progress. “The idea that disengagements give a meaningful signal about whether an [autonomous vehicle] is ready for commercial deployment is a myth,” he writes.

A lot people, including AV experts, agree that the reports are largely meaningless. But in the absence of any federal reporting requirements for companies testing AVs on public roads, they are often acknowledged as being the best dataset we’ve got to track this technology’s progress.

Vogt goes on to blast other companies for effectively pulling the wool over the media’s eyes with flashy demos that prove nothing but the ability to execute a decent test ride.

Our collective fixation on disengagements has been further fueled by the AV companies themselves. Many of them give demo rides that include a de facto story-line that goes something like, “I didn’t touch the wheel during this demo, therefore it works.” This is silly, and everyone knows it. It’s like judging a basketball team’s performance for the year based on how they looked during a practice session. Or winning a single spin of roulette and claiming you’ve beaten the house. A carefully curated and constrained demo ride is just the tip of the iceberg, and we all know what happens if we ignore what’s lurking below the surface.

The AV industry is in a trust race, so it’s important that we do things to build confidence in the technology. It’s certainly convincing to go on a ride where it seems the human is just there for show, or on rides where there’s no human present at all. So companies carefully curate demo routes, avoid urban areas with cyclists and pedestrians, constrain geofences and pickup/dropoff locations, and limit the kinds of maneuvers the AV will attempt during the ride — all in order to limit the number of disengagements. Because after all, an AV is only ready for primetime if it can do dozens, hundreds, or even thousands of these kinds of trips without a human touching the wheel. That’s the ultimate sign that the technology is ready, right? Wrong.

It’s hard not to see this as a not-so-subtle shot at Waymo, which has been giving members of the public, as well as some reporters (including me!), rides in its fully driverless vehicles in its test area in the suburbs of Phoenix. Cruise has long been critical of Waymo’s decision to concentrate most of its testing in a suburban environment, rather than a more complex, urban one like Cruise has.

“Keep in mind that driving on a well-marked highway or wide, suburban roads is not the same as driving in a chaotic urban environment,” Vogt writes. “The difference in skill required is just like skiing on green slopes vs. double black diamonds.”

(Waymo likes to point out that it tests its vehicles in dozens of other cities, including dense urban ones like San Francisco.)

Cruise has been waging a slow-burn war against the California disengagement reports for years now. Jalopnik took the company to task in 2018 for failing to report an incident when one of its self-driving cars ran a red light in San Francisco after the safety driver took control to avoid blocking a crosswalk. According to Cruise’s interpretation of the California requirements, the human driver didn’t act out of safety concerns or a failure of the autonomous system.

It’s hard not to see this post as Vogt’s attempt to get ahead of next month’s disengagement reports. He lists a variety of reasons why Cruise’s safety drivers will routinely take control of their test vehicles, as a courtesy to other road users or just out of an abundance of caution. And he even goes so far as to include a graph that he says shows Cruise’s improving rates of disengagements, or the distance traveled between disengagements.

Vogt says Cruise will lead the charge to release data that more accurately conveys the progress that it is making.

Ultimately, I believe that in order for an AV operator to deploy AVs at scale in a ridesharing fleet, the general public and regulators deserve hard, empirical evidence that an AV has performance that is super-human (better than the average human driver) so that the deployment of the AV technology has a positive overall impact on automotive safety and public health. This requires a) data on the true performance of human drivers and AVs in a given environment and b) an objective, apples-to-apples comparison with statistically significant results. We will deliver exactly that once our AVs are validated and ready for deployment. Expect to hear more from us about this very important topic soon.

Vogt ends by issuing a challenge to his competitors to follow Cruise’s lead in uploading long, uninterrupted footage of their vehicles driving the streets of San Francisco. “Despite all the AV demos that have taken place over the years, the only video that I’ve found anywhere online that shows continuous, rideshare-like autonomous driving is the one that we posted back in early 2017 (slowed down version here),” he says. “I haven’t seen any others that are more than a few minutes long.”

Cruise, which is backed by Honda and General Motors, is gearing up to make a big announcement next week concerning its desire to “move beyond the car.” In preparation, the company emptied out its Instagram account and replaced it with nine posts, each displaying coordinates in latitude and longitude that correspond with automotive engineering history.

https://www.theverge.com/2020/1/17/21070620/cruise-california-disengagement-report-self-driving-car