Waymo has been very protective of its testing process in past, but recently it started opening up – likely as a bid to help get the public more comfortable with self-driving vehicle technology as it moves towards broad deployment of its autonomous cars. As part of that, the former Google self-driving car project asked a group of journalists to pay a visit to its Castle testing facility in Northern California.
The Castle isn’t just a very cool name for a proving ground, it’s the actual name of the former Air Force base (used during the 1940s for training bombers for WWII) that Google took over back in 2013 to house some of its ‘X’ projects, including Project Loon and what would eventually become Waymo in 2016.
At Castle, we got a rare look at one aspect of Waymo’s testing process for its autonomous cars, complete with a briefing on the company’s approach from CEO John Krafcik, VP of Engineering Dimitry Dolgov, UX and Early Rider Program Product Manager Juliet Rothenberg and Head of UX Design Ryan Powell.
Krafcik opened by giving a rundown of the various terms that have been applied to self-driving technology, ranging from “driver assistance” to “semi-driverless cars,” noting that there’s been “a lot of confusion about what the terminology means.” Part of Waymo’s aim is to clear up the confusion – and by implication, perhaps douse the cold water of reality on some of its competition’s more grandiose claims.
It also helps Waymo clearly explain where they sit on the spectrum of driverless vehicle technologies, and how they concluded that they would focus only on technologies that would classify as Level 4 and Level 5 by the SAE’s standards – fully driverless tech requiring no intervention by a human driver.
Waymo classifies anything from Levels 1 through 3 as technically “driver assist” features, according to Krafcik, and this is an “important divide” which Waymo has observed first hand, concluding early on that it’s not an area they’re interested in pursuing.
Krafcik revealed that one of the first products Waymo considered bringing to market back in 2012 and 2013 was a highway driving assist feature, which would handle everything. between onramp and exit, but that also required drivers to be fully attentive to the road and their surroundings while it was in operation.
The results, per Krafcik, were downright frightening: Footage taken from the vehicles of Google employees testing the highway assist features, which the company showed us during the briefing, including people texting, doing makeup, fumbling around their seat for charge cables and even, in one particularly grievous instance, sleeping while driving 55 MPH on a freeway.
“We shut down this aspect of the project a couple of days after seeing that,” Krafcik said. “The better you make the driver assist technologies… the more likely the human behind the wheel is to fall asleep. and then when the vehicle says hey I need you to take over, they lack contextual awareness.”
This is why Waymo has been very vocal in the past and today about focusing on Level 4 (full autonomy within specific ‘domains’ or geographies and conditions) and Level 5 (full, unqualified autonomy).
How does Castle enter into its goal of achieving that by “building the world’s most experienced driver?” In short: Practice.
Waymo likes to quantify its progress in terms of miles driven, since driving experience is the primary means of improvement for autonomous technology, according to it and many others in this space. Krafcik said at the event that Waymo has managed 3.5 million autonomous miles across testing in 20 different cities this year, and it managed 2.5 billion (with a ‘b’) miles in 2016 in simulation, or via testing in virtual software environments reflecting real-world conditions.
At Castle, you get aspects of the unpredictability of real-world driving, combined with the control of simulation. Stephanie Villegas, who leads ‘Structured Testing’ at Castle, explained that this type of testing allows them to model and stage challenging situations Waymo has encountered in real live, and also to validate things they know the cars have been able to do well when they issue updates to make sure there aren’t any regressions.
Structured Testing sounds kind of complicated but it’s actually explained in the name – Waymo sets up (structures) tests using its self-driving vehicles (the latest generation Chrysler Pacifica-based test car in the examples we saw), as well as things they call “fauxes” (pronounced “foxes” by Villegas. These are other cars, pedestrians, cyclists and other variables (contractors working for Waymo) who replicate the real world conditions that Waymo is trying to test for. The team runs these Tess over and over, “as many times as we can where we’re still seeing improvement” per Villegas – and each time the conditions will vary slightly since it’s real-world testing with actual human beings.
Waymo and Villegas took us through three structured tests, including one in which a passing car cuts off the self-driving van without much warning; one where the self-driving car has to deal with a vehicle backing out of a driveway on a corner; and one where it encounters movers in a roadway and has to navigate around them, while also heeding oncoming traffic.
The self-driving car handled every situation, all of which featured three test runs, with aplomb, and Villegas said that although Waymo always performs its tests with safety drivers on board, each of these was done fully autonomously with no intervention from the drivers at all.
In general, Waymo says it’s essentially nailed down Level 4 self-driving, especially in the controlled confines of its autonomous proving ground at Castle. We even got a ride in the self-driving Pacifica – without anyone even at the wheel at all – and it went smoothly (more on that here). But it doesn’t sound like Castle’s useful life is anywhere near at an end: Villegas said that she’s run countless Structured Tests in her time at Waymo since 2012, first at a semi-private and disused parking garage for the Shoreline Amphitheater near Google’s Mountain View HQ, and later at Castle when the company outgrew that space.
Getting Waymo’s tech ready for Level 4 autonomy in public deployment will require more testing still, as will making the eventual leap from Level 4 to Level 5, the company’s true ultimate goal. It’s not just a matter of having somewhere Waymo can test without worrying about state regulations – it’s about a place where serendipity can be manufactured, to help ensure its cars are ready for anything, without having to wait for them to encounter those scenarios on real roads when the stakes are highest.