Autonomous driving is among the most exciting applications for artificial intelligence.
It is also among the most challenging to actually pull off: from the capital-intensive combination of hardware and software to the complex coordination between multiple stakeholders to ensure it is developed and deployed safely.
But the field has made significant leaps forward in recent years, and today there are multiple forms of autonomous driving technology actively being tested on the road including driverless delivery vehicles and software and hardware platforms designed to transform traditional trucks and cars into self-driving machines. But how close are we, really, to becoming a society of passengers?
The answer is closer than many may think, but still far enough off that innovation requires a team effort between technologists, business leaders, policymakers, and society as a whole. I sat down with two leading autonomous vehicle entrepreneurs Aurora CEO and co-founder Chris Urmson and Nuro CEO and co-founder Jiajun Zhu to discuss the current state of the field and what the driverless future may look like.
This interview took place during Greylock’s Intelligent Future event, a daylong summit featuring leading experts and entrepreneurs in AI. You can watch the video of the interview on our YouTube channel here, and you can listen to the conversation at the link below or wherever you get your podcasts.
EPISODE TRANSCRIPT
Reid Hoffman:
I’m delighted to invite two other friends up on stage. The CEO and co-founder of Aurora, Chris Urmson and CEO and co-founder of Nuro, JZ (Jiajun Zhu) two of the folks who I also learned from. Part of our selection of the folks that we would have come to join us are people that we learn from intensely.
Let’s start with the current state of what’s happening on roads and autonomy. I guess I’ll start with you Chris. What I think most of the people in the room will understand is that actually, we’re already on the road. But they may not realize how much of it is actually already pragmatic. And I kind of got that because of sitting in the truck, going on that path from Dallas towards Houston, looking at the server closet, looking at all the rest going, “Oh my god.” You could see it. Say a little bit about what is happening today.
Chris Urmson:
Yeah, I think one of the things that’s happened in the self-driving space is, over the last few years we went from a mode of, “Hey, it’s all solved. It’s done tomorrow, we’re going to ship it next week,” to, “Boy, is this ever really going to happen?”
And I think both extremes were and are wrong. That today there are vehicles on the road, you can get a ride in a cruise vehicle, you can get goods delivered by Nuro, you can get a ride in a Waymo vehicle with nobody behind the wheel, you can come down to Texas, you can see one of our trucks driving between Dallas and Houston.
So this is happening today, it’s happening in a meaningful way and we’re pulling loads. If you got a FedEx parcel that got shipped from Houston, there’s a good chance that it sat in one of our trucks on the way there, to you. So I think it’s incredibly exciting, it’s going to take a little while as we kind of do things in the physical world, and we put in place the supply of vehicles and integrate with them and do all the work that we have to make sure these things are safe, but it’s coming along really well.
RH:
Yeah, we’ll come back to safety in a bit. But also, JZ, same question, in part, we have one of the vehicles here that actually has California tags on it.
Jiajun Zhu:
Yeah.
RH:
That kind of rolled in, but same thing, what does today look like, looking out for the next couple months?
JZ:
Yeah, definitely, I completely agree with Chris. I still remember probably 13 years ago we started working together on self-driving [tech] I mean, it’s been a long journey. I think this is definitely already zero to one, we’ve passed the one, now it’s about scaling.
So as Chris said, we already have self-driving cars on the roads, different companies have different services. Last year, we had the first driverless pizza delivery in Houston and it was just amazing to see customers ranging from a 3-year-old all the way to 80 years old using a robot, getting their pizza from Domino’s, so this is definitely happening. I think now the industry is trying to figure out how to grow, how to put more vehicles on the road safely and grow this as a business.
RH:
So part of what makes us at Greylock just amazingly delighted investors in both companies (and which was part of our discussions early on), is that you were both very sophisticated about how you approached safety, because obviously safety is one of the important things.
I’ll just invert it just so we’re not always doing the same pattern: I’ll start with JZ and then go to Chris. Just talk a little bit about how safety is fundamental to your architecture from ground up and what you’re doing.
JZ:
Yeah, maybe I’ll talk a bit more about our unique application. We focus on goods transportation, there’s no passenger in the vehicle. And by not having a passenger vehicle, we get the opportunity to really design the vehicle from the ground up to optimize for safety for other road users.
So not only do we try to design a vehicle that is very neighborly friendly, but also it’s a bit narrower, so you have more space around you, you can avoid accidents more easily. And because there’s no passenger, you can do things on the vehicle that you cannot do on a passenger vehicle. So for example, at the front end of the vehicle, we have external facing airbag, so this is in case there’s accident, there’s a collision, the vehicle will detect that and deploy the airbag 30, 40 millisecond right before the collision, so this is something that we think is going to be really, really good for safety.
Another example, also, because there’s no passenger in the vehicle, we can decelerate the vehicle much more aggressively than you can do if you have a passenger inside. So these are examples of things that we had basically at the first day when we started designing the vehicle, trying to leverage the fact that we don’t have passengers inside.
CU:
And for us, we drive passengers and we drive giant trucks, so we’re neither skinnier nor can we stop as aggressively and airbags from the front end. Well, I guess we could, but I don’t know that it helps with a 70-ton truck or whatever we’re driving down the road.
And so for us, we have to think even more holistically about safety, we embed it right in our culture. If you look at our company’s mission is to deliver the benefit to self-driving technology safely, quickly, and broadly because it was so important to put it in place up front.
And then we’re one of the few companies, I think maybe the only company, that has actually shared how we’re going to convince ourselves the thing is safe to be on the road. And this big structured argument: we call it a safety case framework that breaks down the technical reasons when things are working, why they’ll be safe, the technical reasons why when things break, we’d still be safe and then all the organizational and process stuff that we do around that to help make sure that the system’s constantly improving and that we have controls in place. And so we really just don’t think you should be doing this without doing that hard work. And as we look at this technology, it’s going to shape transportation for the next century. And so investing in that foundational part of this so that we don’t kind of foreclose the future, I think is critically important.
RH:
Yep. Now related to safety is regulatory, and regulatory has good and regulatory has bad. What are the places that each of you would kind of suggest/nudge/say, “Here is the kind of regulatory environment that governments (especially US, especially California), we’ve all interfaced with, would say, “Do more of X, do less of Y”? Because both of you were totally in favor of safety regulation is good for that, et cetera, you’re pro-regulation, but how to do it intelligently is a thing. So, JZ, do you want to start?
JZ:
Yeah, we engage with the government very, very early. Even before we came out of stealth, we were working with the Department of Transportation. And I think there has been a lot of success on that engagement, so we became the first company with a vehicle that got the exemption from DOT. It doesn’t have a steering wheel or a brake pedal, but it’s road legal. We also got the first commercial deployment from California, which is great. We’re also the first driverless pizza delivery in Houston, so to us, our mission is to better everyday life through robotics.
So it’s really, really important for us to really work closely with the policymaker to help them see the benefits. And I think, so far it has been well received. There are areas where we think we can potentially move even faster, so I think maybe not on this vehicle, but if you look at other vehicles on the road, we still have a tiny little side-looking mirror. And this is one example where we still have to work according to the current regulation and to make these vehicles really follow the current code. But this is not necessary for autonomous vehicles. And I think these are areas where policymakers could really help us.
CU:
I think when it comes to regulation, it’s important to understand the landscape as it is today. In the vast majority of the US, if you have a vehicle that you’re confident in the safety of, you could put it on the road and operate it without having to do anything else. And so that is a real competitive advantage for the country.
When we talk about regulatory change, it’s really around the edges and it’s to help maintain that advantage over time. And so much like Nuro, we’ve been engaging both of the federal and state levels since day one. And it’s about helping the regulators, helping the lawmakers understand the promise of the technology, understand really what the challenges are with the technology so that they can be informed, because the worst thing that happens is something bad happens – and it will, right? We’re talking about the real world. [When that happens], and some congressperson calls the regulator and they have to make a knee jerk response at that moment versus when they’re informed, they understand the situation, they understand the issues, and they can respond in a thoughtful way. And so that’s been our strategy over time.
As I think about what could be better, we would love at some point when the societal benefit of this technology shows that you get protections for litigation that look like the airline industry, right? That would be great for business, great for society, we would love to see more nationalized, harmonized regulations.
So the way the regulatory stack works is the federal government regulates the safety of the vehicle. The states regulate the safe operation of the vehicle. Normally those are relatively deconvolved. Obviously when you build a self-driving vehicle, they become quite convolved. And so it’d be great that the rules of the road, the safety expectations were consistent across the country.
RH:
A lot of folks in the room will already know (because they’ll have some ties to the AI world and so forth), what the importance of simulation is and how do you wrap that into your process of learning, and all that, so we don’t go into that.
Let’s actually go a little bit to the tactile hardware parts of it. One of the things I and both of you know very well and we, having collaborated on this, there’s a little bit of a debate of, “Oh, you should just do it with cameras,” and “No, no, no, actually do it with Lidar,” and all the rest of these things.
And actually in fact, my own point of view, which I think is, I’ll state it for myself, but I think it’s reflected in the things you guys also believe is like, “Well, why not have maximum safety?” If I was driving the car and I had lidar too, that’d be great because let’s make it safer. Sure, I have perfectly functionalized and responsible, let’s make it safer.”
Say a little bit about the kind of sensors, sensor fusion, the approach for making this not just as good as what we have, but so much better than what we have. JZ, I’ll start with you and then Chris, I’ll go to you.
JZ:
Yeah, sure. I completely agree with you Reid.
So we have lasers, we have radar, we have cameras, we have thermal cameras. Some of the sensors are built in-house and we really believe that this is the right pass for Level 4 autonomy. And I think maybe one insight was that I think people tend to look at companies like Tesla, “What is their approach?” And I think there is a little bit of nuance here.
I think the optimization function, if you think about building a product for a consumer vehicle versus building a fully autonomous fleet, maybe you operate the fleet, that’s your business. The optimization or the objective function could be very different. I think the cost of the sensors could be different. [For example] how fast – and you’re going to use all of these sensors – how fast you can replace them. And I think all of these things become very, very different.
So I think for our business where we operate a fleet of autonomous vehicles for goods delivery in this case, we really believe that using the most advanced, redundant sensors to maximize for safety is the right path.
CU:
Yeah, so we’re going to break news today and we’re going to announce that we’re ripping everything that is in cameras off of the Aurora vehicles. Now, we totally don’t believe that, right? We agree completely with the way Jiajun thinks about this.
I can’t remember who said it from the Tesla camp, but they said lidar sensors are a crutch. And in my mind or my model, engineering’s all about cheating, and so anytime you can cheat by using better capabilities, better technology, you should do that. It allows you to solve the problem more quickly, it allows you to solve the problem in better ways.
And again, as we think about whether it’s a multi-thousand pound light vehicle or a tens of thousand-pound truck driving down the road, having redundancy in the way you understand the world, having the robustness that comes from seeing it, different frequencies and thus different penetration of weather, different will face things fail. And it’s really the only way that you’re going to get something robust enough that we could really trust it.
I am one of those people who bashes human driving – 40,000 Americans killed last year and that number’s going up, but the flip side of it is, it’s something like one for every 85 million miles of driving, which is a really impressive statistic. And so if we want to hit that level of performance, that many nines of reliability and robustness, we have to pull all the tricks out of the bag to be able to do it.
“There’s not a whole lot of point in scaling something if it can’t do something useful in the world.”
RH:
Well, that kind of actually led to the next question that I had here, which is what should be the expectation of the benchmarks of the OKRs of what is safe enough, what is efficient enough for full “pedal to the metal” we should be deploying the stuff as possible? We already know that we have enough safety on the road, we’ve got various things happening in Texas and other places. But what should be some of the concepts where people say, “Okay, that’s the measure,” because it shouldn’t be [this notion of] zero accidents. That’s foolish.
CU:
Yeah, right.
RH:
It’s irrational and self-destructive, so what should be the way that people think about it?
CU:
So first, it’s got to be commercially relevant. There’s not a whole lot of point in scaling something if it can’t do something useful in the world, and I think we’re both fortunate that we’re working on applications where there is immense value. We have a tremendous shortage of drivers, we’ve just seen the impact that [issue] led in part to the supply chain challenges, so there’s got to be both commercial and social value there.
And then on the safety side, you’re right, we can’t let perfect be the enemy of good, right? And over time, this technology is going to improve as we get more experience on the road. It’s going to improve, we’re going to learn more about how things fail. And so we need to find an acceptable level of risk that is not unreasonable.
And by the way, that’s the bar that the Department of Transportation uses that you see – that you are not creating unreasonable risk on the road. So for us, we think about how do we know the Aurora driver’s going to do the correct thing? And I would posit for you, imagine you’re driving down the road and a vehicle swerves into you. This is actually one of the very common ways that trucks get in an accident. It’s actually another vehicle swerving into them. Well, it’s probably not okay for you to be oblivious to it and just have them hit you. You should probably take some kind of responsive action, but it’s also probably not okay for you to have to drive off of the road to avoid it, right? There’s a lot more risk that comes with that, so there’s some correct behavior which is to move away up to a limit and slow down.
So as we think about it, it’s kind of spelling out some of those underlying simple, understandable guiding principles and achieving a level of performance that’s comparable to a good human driver.
JZ:
Yeah, the only thing that I would add is one thing that I learned from building Nuro (and also previously working with Chris at Google) was I think building the autonomous system that is in a real-time, safety-critical environment in the physical world is really, really hard. And in order to do that, it’s not just the AI part needs to be really, really good, but you also have to develop this really amazing foundation in the company from requirements all the way to validation to go through the entire process that allows you to have the confidence in the safety performance of the product that you’re building. And I think that is a pretty unique aspect of autonomous vehicle technology.
And as Chris said, with which I agree, is not to introduce unreasonable risks when we deploy. How do you actually measure that? How do you actually quantify that? I think this is really, really the challenge that we’re all working to solve.
RH:
So I’m going to ask one more question and then try to leave room for two audience questions versus one, this time.
So what are some of the most challenging parts of the environment that each of you is navigating? Which is the part that makes you say, “This is why this is here, and why now,” but also the result of so much work and the work that you’re still doing.
One of the things I thought was funny when we did the investments was watching how easy it was to get someone going to 70% and then the rest of it started getting much, much harder. And both of your guys focus on that, both of you taught me things around, “Okay, how are you dealing with construction cones? How are you dealing with tunnels? How are you dealing with…” Those are the questions I started asking people.
CU:
So, with us, as we think about trucks driving at freeway speed, the amount of kinetic energy involved is large. And so to be able to kind of slow down and avoid things, you have to look out, getting on for half a kilometer, 400, 500 meters. And it turns out that’s hard, right? You either have very narrow field of view cameras or preferably you have narrow field of view cameras plus amazing sensors. And in our case, we’re developing this special technology, FirstLight, which is a kind of lidar that can see further than the pulse lidars that almost everybody else uses. And it allows us to both see further, it allows us to instantly measure how fast things are moving so that we could figure out how much of a threat they are. And that allows us to react meaningfully sooner. In the case of what we’ve been looking at for our sensor versus some off the shelf lidars, we see about two football fields further than they can and that is about seven seconds sooner that we get to react, which is just magic.
JZ:
Yeah, I think the long tail edge cases are also evolving over time. So I think maybe 10 years ago it was very much focused on perception problems. I remember, we probably spent a lot of time dealing with leaf blowers or fog or just reflective science. I think nowadays, I think the edge cases for us, we focus a lot on low speed neighborhoods, maybe the opposite end from the high speed road where you have to see very far, we don’t have to see very, very far, but you have to be able to handle a lot of unpredicted agents in the world. So for example, a kid who is riding a bike – that could be very, very unpredictable. Kids running into the roads, chasing the ball, these tend to be some of the hardest cases.
RH:
Right.
[To audience] Questions? And if not, I have a ton.
Audience Member:
When are we going to see self-driving cars in business on public streets in America?
RH:
Thanks.
CU:
Sorry, when are you going to see self-driving cars? Today. You go to San Francisco today and you can get in, ride around and enjoy it.
RH:
Actually, do you want… Go ahead and refine.
Audience Member:
Part of the question was, as a business, when we’re going to see them commercially deployed. I know right now it’s in the testing phase, and I think they’re not available to the general public yet, and some of them at night have a driver inside. Some have no driver inside, and that’s rare.
CU:
Yep.
Audience Member:
When are we going to see them as a business?
CU:
Yeah, so one of the reasons why Aurora’s focusing on trucking is because it’s the business we think makes more sense earlier, and that there’s all kinds of really great social reasons. But you’re asking about business: a truck driver, we pay about three times as much as we pay a ride-hailing driver, and we’re short about 70,000 of them. So there’s a real economic need and we expect that driving on freeways. It’s much more similar than driving through different neighborhoods and so we expect there to be kind of operational scaling rather than technological scaling.
For us, we’re looking to launch, we’re working towards the end of ’24 where we’ll have fully redundant trucks. That point, the trucks will be scalable and we’ll expect to start building a really exciting and interesting business at that point.
JZ:
Yeah, we are already doing commercial deliveries, but at a small scale. If we are talking about more scaled commercial operations in multiple cities, I think this is going to happen in the next two to three years for us.
RH:
Yeah, I think this is all within a couple years, but you partially know the answer to that question so…
Audience Member:
Well, that’s your perspective.
RH:
Yes, another question and it’s all right if no, because I will ask one. All right, I will ask the last question then.
So what do you think this plays to globally? The very natural thing, is of course we have a whole bunch of intensive things playing here in the U.S. but there’s a question about, obviously there’s a bunch of stuff going on in China, there’s a bunch of different environments in whether it’s India or Africa, so we’ve got all of this intense work working within kind of the first world. How does this broaden to as it were crossing the digital divide?
JZ:
Yeah, that’s a great, great question. I think there are probably going to be multiple systems. I think it’s not going to be one winner take all, I think for many, many reasons. One is that this is taking a very, very long time to develop. It’s very hard for the first one to have an advantage way bigger than the second one. I think there’s also political reasons, for example, the U.S. and China, I mean there’s national security reasons why for a U.S. company, it probably will be very hard to go map Chinese roads and then vice versa. But I don’t think it’s going to be 20 companies. I think it’s probably going to be a very small number of companies in the U.S., maybe Europe, but maybe there are also a handful of companies in China. And then I think we’ll grow into other countries.
“There will be multiple [autonomous driving] systems. It won’t be winner-take-all.”
RH:
And before we get to yours, because this to reflect is one of the points I’d learned from Sebastian [Thrun], years ago is that all of these vehicles out there are learning machines and they’re sharing their learnings. So when you have it in the same network, even if it’s in different geographies, you get improvement from that. And that’s part of the not “N of 1”, but a small N will be the number.
CU:
Yeah, I also think it’ll be a small end. I think it’s going to be balkanized for the same reason Jiajun does. We’re seeing it on the internet, this is going to be more scary to governments than moving data, right? We’re moving atoms through the world with a lot of energy, so I think that’ll split China and a lot of the western world. I expect it’ll be kind of like the U.S. because of regulatory flexibility, Europe possibly like Korea and Japan just because of the population demographics is going to be important there. And then it’ll, I think two decades from now, maybe this looks like what happened with mobile networks instead of landlines. That if you are developing new cities, this is what you build the city around is automated transportation. And so that may actually be interesting and create a leapfrog there as basically the cost of the systems come down over time.
RH:
Yeah, one of the things that I think is also very interesting that we’ve all talked about is how much each of these vehicles now becomes, now, a data center of inputting data. All of the things that could mean for how you make society much better when you like, “Oh, I saw a person collapsed at the side of the road, maybe we should call medical services.” All of this stuff is part of the lens ahead of this, that part of why we at Greylock are so delighted to be partners of both of you is that you are thinking about these questions. [Things like] “How does our vehicle massively improve society? What are our responsibilities on the downside like safety, but also how are people’s lives so much better, including work from truck drivers?” and all rest.
So on the behalf of all of us, thank you for joining us.
CU:
Thank you.
JZ:
Thank you.
