Waymo's EMMA: Teaching Cars to Think - Jyh Jing Hwang, Waymo

So I think a lot of people here already took Waymo here in San Francisco, so we drive every day, and I think the technology just works, as you all witness. So I'm going to talk a little bit about the history of autonomous driving research, like what it took to take us here, and in the current status.

So in the autonomous driving history, people started the research in the 1980s, where it's just very simple neural networks, there are three layers, and over time people start to think about, you know, going deeper and deeper. And then in around 2020, there are papers from NVIDIA and other research labs publishing end-to-end driving models.

And this is one of the videos that they published at that time,right, and then we can check it out,right. The car in front, driving, is the autonomous driving car from the research labs at that time, and you can see that it's actually drifting,right.

So this is more like an O2 sort of technology, and you wouldn't want to basically ride in it. So why is an L4 system that Waymo has so much better,right, you can see it drives around San Francisco in various regions, in downtown areas, and it avoids pedestrians, cyclists, and it makes roads very safe.

So the secret is very basically on the top, where you can see that this is a sort of visualization or debugging screen, where it visualizes everything that the system understands. So it can actually capture all the cars, pedestrians, cyclists, traffic lights, crossroads, and it understands almost everything related to driving.

And we can see this is a very complicated system. If we break it down, it's basically a perception system where it understands the world, and there is a prediction system that basically predicts the future of future world states, given the current one.

And finally, there is a planning system that basically tells us, like, how we should drive in this scenario, like we should turnright, how much acceleration, how much steering, etc. So this is the whole autonomous driving system that is very complicated and delicated.

But it solves problems,right, it drives in San Francisco today, so what else? So what are we heading with autonomous driving? The answer is scaling,right. So we've proven that we are operating in Phoenix, San Francisco, Austin, and Los Angeles today, and it's offering the rider-only service.

However, we are our ambition is not going to stop in just four cities now. So in this year, we are doing a road trip where we are visiting about 10 cities in various locations. Here is the map, and you can even see that we are actually going across the globe to Tokyo, Japan.

So there is a lot of challenges when we start to scale.

So just recently, we finished an open challenge for participants to solve some of the hardest problems that we see in our spending. So let's replay the video a little bit here. So on the top left, there is a marathon run on theright, and then there are cones around the roads.

And most interestingly is that when we drive closer to the intersection, if you look at the traffic light, it's actually red, but there is a traffic control man waving us to go ahead. So it's actually green. And this is very confusing,right, and all these long tails are very challenging to solve.

And there are challenges in, like, a lot of times that you've never seen it in your entire life of driving. But for Waymo, at Waymo's scale, we will see it very, very often, and we need to solve them in order to scale.

So one of the solutions arises where foundation model is very generalizable,right. So let's just take an example here that we feed in Gemini,right. You can see the video where when we were driving peacefully, suddenly there are a bunch of angry birds,right, they appear and attack our car virtually,right.

So that's where we need to react. But for a car to understand the scenario, it also has to understand, okay, this is a fleet of birds, and how they are going to behave, how they are going to interact with the car,right.

This is not the everyday driving that you see, but this is a critical long tail. So let's see what Gemini responds. So Gemini basically says that a large flock of birds suddenly takes flight from the ground in front of the vehicle, and the expected behavior is basically to slow down and remain alert, and possibly adjust the speed afterwards.

So this is a very, basically, I would say, perfect answer to how we should drive with respect to this scenario. And this is another scenario where you can see there is a scooter rider in front of us, and unfortunately, she slipped,right.

And this is because basically it's at night, and there was rain earlier, so the road was wet. So these kinds of events also happen quite often, like when the scale is large,right. But this is very much a safety-critical event that we really want to get itright and get it perfect.

And let's ask Gemini again. And Gemini actually identified the entire scenario correctly, and even identified a gas station in a faraway scene. So I think it's very telling that in today's foundation models, they actually generalize well in these kinds of rare events.

And the question becomes, like, how do we, like, how do we leverage this technology for autonomous driving? So here is our exploration in research, where we want to have a more generalizable autonomous driving system by leveraging Gemini or other multimodal large language models.

So here we call it EMMA. So this is the simplest form of EMMA, where the idea is very straightforward, yeah. Gemini is very good at generalizing in various scenarios, so let's just put it to drive. So how do we do that?

So on the top left is about router, where we want to know where we are going in this current driving. So for example, like, you can think about it as a Google Maps signal, and then it tells us, you should turn left or turnright at the next intersection, something like that.

And then we translate it into text, just pure text. And then on the bottom left, we have the vehicle with surrounding cameras. There are eight cameras in this image, and it covers 360 degrees. And then we just input this video and the text, the routing text, to EMMA, basically built on top of Gemini.

And then we ask Gemini how to drive in the next scenario,right. So Gemini will output future waypoints, which means the future locations of this car should be in the next few seconds. So this method is very simple,right, like, and there are three major traits here.

The first thing is that it is self-supervised, because whenever we have a driving log,right, we know where the car were. So at any time point, we can use this formulation to train the model, because we know where the future the car should be in.

So this is self-supervised, very scalable. The second thing is it is camera-only. We don't have LiDAR yet, because, you know, Gemini is a camera model. So we only need cameras to drive. The third thing is that it is high-definition map-free.

I think for people who understand autonomous driving technology to a certain extent, like, we usually hear that Waymo's car needs a lot of map priors. So in this model, we actually don't need any maps besides Google Maps. So how was the performance here?

So we basically conducted a research benchmark called New Things. This is one of the most popular open-loop planner benchmarks. And for this simple model, it achieves state-of-the-art quality compared to all other models at that time. So basically, whatever, like, customized models, small model, large models, like, with this simple formulation, we can already achieve the best quality.

So now we are thinking more towards, like, okay, so the self-supervised method does work to some extent, but there are a lot of drawbacks that we want to basically remedy,right. The first thing is that, okay, we have had a lot of labeled data, a lot of experimental models.

How do we leverage them to further improve the quality? The second thing is that there is a very clear drawback about end-to-end models, where the explainability is not there,right. Like, we can only see the output of planner, but we don't know what happens inside.

So here is what we do,right. We basically have a channel process before outputting the planner. So basically, we let the model explain itself about how it should drive in this scenario. For example, we ask the model to identify critical objects on the road first.

In this scenario, they identify the cyclist and the vehicle. And then we ask them to explain what kind of behaviors those critical objects will do, and what kind of driving meta-decision that we should go for. And in this scenario, it says, we should just keep the normal speed.

In some of the scenarios, it will say, we should yield or slow down, something like that. And with this, we actually achieved an even better planner. It's measured in our own Waymo open motion dataset. And this dataset is actually 100K dataset, about 100 times larger than the New Things dataset.

And more importantly is that the baselines are much stronger, because they are not, they are specialized models. They are waveformers and motion LLM. So they are basically built on top of very sophisticated Oracle perception system, and they take inputs from those Oracle perception and road graph, that's basically high-definition map, and traffic light states.

So they have inputs from everything almost, and then their own job is to output a planner. So these are very strong baselines. And with channel reasoning, you can see that it actually comes on top. So with this architecture, we actually see that it actually performs very well in the academic setting.

So one promise in foundation model is that it openly agreed, it kind of popularized the scaling laws. It basically says that the quality will keep improving if you have a larger model and larger dataset. And we are showing here that if we train, like, this is a dataset that's magnitude larger than any of the academic datasets that's released publicly.

And we see that if we keep training more data, and we see the quality keeps improving. So the y-axis is the perplexity here. So it basically means that the lower the perplexity, the better the planner results. So we see the quality can be further improved.

So we talk about the explainability here, and one thing that we also are thinking about is that why not just train on various tasks,right, like, because it is a vision language models at the core, and the language part is super flexible.

We can basically formulate any types of tasks with it. So we think about, like, let's make EMMA the most generalizable model possible,right. So we add a lot of different tasks into it. In this example, we demonstrate 3D detection, road graph estimation, and some kind of free-form VQA here.

And for any prompt on the left, if you enter it, then EMMA will spit out the answers correspondingly. So it will have the first one is the end-to-end driving, the second one is 3D detection, etc. And then we decode and visualize the results on theright.

So we also measured the detection quality here on our Waymo open dataset. So it actually achieves very similar quality compared to other state-of-the-art models. So this demonstrates the EMMA's capability generalizing to different tasks and co-training them together. So here are some visualizations of the EMMA's outputs, including the driving trajectory detection and road graph.

So I think the predictions are reasonably well.

So now it seems like we have some kind of prototype for end-to-end multimodal large language models. But then the question comes next is about how we evaluate it, validate the version, and make sure that it's safe. So evaluation is part of, you know, the entire solution,right.

We cannot really make this model succeed without any evaluation. And for evaluation, previously, all the results I showed was about open-loop evaluation, which means that we just replay the video and then check the model quality. But this is usually not the most faithful way to do evaluation.

So there are simulations and real-world testing. And simulation means that we create a virtual world where we can test our model to drive the car in that world. And real-world testing is just deploying it. So the difficulties are different, and usually we trust simulation a lot more compared to open world.

So one thing that we are also thinking about is that, yes, foundation models in the generative world is also very advanced. This is a VO2 from Google DeepMind, where we see the videos generated are very realistic. So we are also thinking about, maybe we can leverage it for simulation.

So this is also, like, our latest research, sensor simulation for end-to-end driving evaluation. So for this model, we actually use open-sourced generative models from Google, and then we generate the videos, and then we use that to basically place our EMMA, and then we can evaluate the quality.

And we can also control in various conditions, like, we can change the weather, we can change the time, we can change a lot of different ways. Due to time, I basically removed those videos, so sorry about that, the visual part.

So this is the results where we can test our EMMA planner in various conditions,right. We can switch from rain to no rain, we can switch different time in day. And this is very much like

aligns with our intuition, where when it rains, when the weather is bad, then usually our planner gets a little bit worse, because it's a camera-only model. So it affects the camera domain. And then for time of day, it also kind of aligns where at night, usually the quality is worse compared in the daytime.

So usually at noon or in the afternoon, the model will perform the best. So this is the last slide. So these are all generated videos that we test our models in. So I think this is an ongoing research, where I think it's a very exciting field, where we have all the foundation models.

And because Waymo and Google both belong to Alphabet, so we get access to all these models for free, sort of. Yeah, so we basically try to adapt them and try to see if we can improve the generalization and help Waymo to scale to the next big thing.

So thank you. Thank you for your attention.