Welcome to Simulation City, the virtual world where Waymo tests its autonomous vehicles

A light gray cube with a thin blue top glides down a darkened highway, beset on all sides by dozens of green cubes. The green cubes bounce between lanes in an attempt to pass the gray cube, but the gray cube maintains a steady speed as the blackened landscape slips past into the artificial night. 

This is Simulation City, the virtual world where Waymo, an offshoot of Google, tests its autonomous vehicles in preparation for real-world experiences. The gray cube with the blue top represents one of the company’s autonomous semi-trailer trucks, while the green cubes are all the other vehicles on the artificial highway. 

Waymo is unique among autonomous vehicle operators in that it has not one but two simulation programs it uses to train its vehicles. The first is CarCraft, which has been in use since at least 2017, and in which Waymo says it has driven over 5 billion miles. Simulation City is the latest virtual world in which the company trains, tests, and validates its “Waymo driver” software in order to ensure its vehicles are better prepared to meet all of the challenges of the open road. Waymo is sharing details about Simulation City for the first time exclusively with The Verge.

The company decided it needed a second simulation program after discovering “gaps” in its virtual testing capabilities, said Ben Frankel, senior product manager at the company. Those gaps included using simulation to validate new vehicle platforms, such as the Jaguar I-Pace electric SUV that Waymo has recently begun testing in California, and the company’s semi-trailer trucks outfitted with sensing hardware and the Waymo driver software. 

The company has also been using Simulation City to run tests in new operating domains to better prepare its vehicles to launch in new cities. Right now, Waymo has vehicles operating autonomously in the suburbs outside of Phoenix, Arizona, and in San Francisco and Mountain View, California. The company also has manually driven vehicles gathering mapping data in Los Angeles. To date, Waymo said it has tested its vehicles in over two dozen cities, most of which are in California. But expansion has been going slowly, and Waymo is hoping that Simulation City can provide help as it seeks to expand to new locations. 

“The way that I would hope that this platform evolves is such that you could get a sense of the rates of different incidents and events inside a city that we haven’t driven in before, before we have to put boots on the ground in that city,” Frankel said, “and get a read on whether the Waymo driver is good or not already for a particular city — and being able to do that for lots of lots of cities.”

Simulation City is well-suited to gaming out “end-to-end” robotaxi trips, such as a 20-minute “rider only” trip across San Francisco. “Rider only” is Waymo parlance for autonomous vehicles without a human driver behind the wheel. The company has approximately 600 vehicles as part of its fleet. More than 300 vehicles operate in Arizona in an approximately 100-square-mile service area that includes the towns of Chandler, Gilbert, Mesa, and Tempe — though its fully driverless cars are restricted to an area that is only half that size. 

“There’s a set of things that we needed to have better tooling to support,” Frankel said, “and Simulation City filled in that grid of capabilities that we needed, and makes it possible to do what was challenging before with the existing simulation tools.”

Simulation is a critical piece of the puzzle for autonomous vehicles. These programs allow Waymo’s engineers to test — at scale — common driving scenarios and safety-critical edge cases, the learnings from which it then feeds into its real-world fleet. 

The key word is “scale” because these simulators allow Waymo to far exceed the distances its vehicles travel on public roads. As of 2020, Waymo said it’s simulated 15 billion miles of driving, compared to just 20 million miles of actual driving. 

In Simulation City, those real-world miles are now informing the miles driven in simulation, meaning the company has more confidence in the validity and reliability of the virtual situations it constructs for its vehicles. 

“Once that relationship is established in an increasingly strong way, we need fewer additional miles driven in the real world to basically say what we learned in simulation is correct,” Frankel said. 

Simulation City is also computationally more advanced than Waymo’s previous virtual world testing in the level of detail it can create. For example, Waymo’s engineers can simulate something as small as raindrops or as complex as late afternoon solar glare. In the past, these situations have been known to confuse an autonomous vehicle’s perception hardware, which can make it difficult to read critical signage like traffic lights. 

In addition to its own real-world testing, Waymo is using a variety of data sources to help build out the driving scenarios for the simulator, including the National Highway Traffic Safety Administration’s crash data systems and the Transportation Research Board’s naturalistic driving study data. 

The company is also using artificial intelligence and sensor data collected by its self-driving vehicles to generate realistic camera images for simulation. They call this technique SurfelGAN (surfel is an abbreviated term for “surface element,” while GAN stands for generative adversarial network, a machine learning term). 

SurfelGAN was created out of a realization that current sensor simulators often use gaming engines, such as Unreal or Unity, which require manual creation of environments and other objects like cars and pedestrians. Waymo’s engineers discovered that these gaming engines were hard to scale and often failed to produce realistic approximations of camera, lidar, and radar data without a ton of extra work. 

So they developed SurfelGAN, which uses texture-mapped surface elements to reconstruct scenes and camera viewpoints for positions and orientations. The use of artificial intelligence helps scale the product, making it easier and more efficient to develop. The technique was outlined in a recent paper co-authored by Waymo researchers, including principal scientist Dragomir Anguelov.

There is a risk in using these new simulation techniques, said Huei Peng, professor of mechanical engineering at the University of Michigan and the director of Mcity, the proving ground for autonomous vehicles. The main one is that flawed input data will produce entirely useless results — known in computational science circles as “garbage in, garbage out.”

“You have to do some kind of correlation to say, ‘My simulation is not junk, there is a very close correlation between what my simulation tells me and what the real testing tells me, even if they are not quantitatively 100 percent,’” Peng said. 

But Frankel says that Waymo has already improved upon what Anguelov and his co-authors outlined in their paper last year. “The maturity of the sensor simulation has grown quite a lot since the time that we published that,” he said. 

Waymo is ramping up its simulation work at a rather precarious time for the company. Earlier this year, John Krafcik announced that he was stepping down as CEO of Waymo after helping lead the company since 2015. Experts say that Waymo still has a commanding lead in the pursuit of autonomous vehicles, but diminished expectations about the future of self-driving cars are affecting its valuation. 

Of course, these are business problems, not technical ones. And earlier this year, Waymo showed how simulation can be used to make a better case to the public for its autonomous vehicles. In a bid to prove that its robot drivers are safer than humans, Waymo simulated dozens of real-world fatal crashes that took place in Arizona over nearly a decade. The company discovered that replacing either vehicle in a two-car crash with its robot-guided minivans would nearly eliminate all deaths.

Waymo has been conservative in the rollout of its robotaxis, refusing to expand the service area in Arizona and holding back on launching a similar service in the Bay Area until it’s sure that its software is up for the task of a much denser and more complex urban environment. 

But Frankel said that Simulation City has helped contribute to the growing “maturity” of its autonomous vehicles that will better prepare them for wider-scale use in the near future. “It’s a level of maturity that I think it’s worth picking our heads up and saying, ‘Hey, world, we’re doing this cool new thing,” he said.