Autonomous Car Masterminds Converge at Google

Stanford University Professor Sebastian Thrun led the team that built Stanley which won the DARPA Grand Challenge in 2005. Two years later, Christopher Urmson of Carnegie Mellon University was the team leader of the group that made Boss that won the DARPA Urban Challenge. And there’s Anthony Levandowski who had the robotic motorcycle in the first Grand Challenge and then went on to develop the unmanned Prius (Pribot) that drove on public roads to deliver pizza.

So how in the world did all three of them get together to develop the Google Car?

GetRobo sat down with the three of them to find out. Here is an edited transcript of a very informative conversation. (This interview took place on Oct. 28, originally for a column on the Wall Street Journal Japan. )

 

(Photo: From left, Sebastian, Anthony and Chris with the self-driving Prius at Google)

 Q. When and how did this project come about?

 S) It started about one and a half years ago here at Google. There was a general awareness of the topic of self-driving cars that came with the Grand Challenges. The Grand Challenges set the stage to communicate to the world, including Google, that there is some interesting technology that may be worth nurturing.

Anthony and I had been working on Street View and that really helped Google as a company build up the next technology and scale up the operation. So there was a general level of trust between Anthony and me and Google as a company. And the key moment was when we got together to discuss what the fundamental things that can shape the 21st century are. What are the big technological innovations that might not feel like immediate business tomorrow morning but where a company like Google should really place a bet on? Google’s mission is really to advance the technology for the better of humankind and (self-driving cars) fit that vision somehow. So the Google leadership basically decided that this is worthwhile to place a bet on and to push it with a sizeable but still a modest investment - good enough to really understand that this technology has potential.

And I like this because it really allowed us to bring together a set of people and a team and a operating procedure which was not possible at the same scale at an academic institution. So I would claim the progress that took place in the last 18 months at Google dwarfs the progress that took in my own lab.

Q. I read that there are 15 people on your team. How many of you are from universities?

 C) There’s Sebastian, me and James Kuffner (who left his position as associate professor of CMU to join Google).  I am on a sabbatical until next June. 

S) I am also on leave. My leave ends next March. Mike Montemerlo (who was the software lead on the Stanford team) now works at Google too. 

Q. All three of you had your own projects. How did they all converge? 

S) When the vision was set (for Google to pursue developing a self-driving car), I essentially made a couple of phone calls to the strongest engineers in the world for this kind of thing. The first person I called was Chris. The second person I didn’t have to call, because we already worked in the same office. 

A) At some point we all competed, but while there is a little bit of rivalry still, we’ve really gotten along. 

C) We competed in the past but we all shared a common vision in terms of where technology was going and we recognized each other’s strengths and we feel the real value in collaborating. In fact we’ve been talking since the second Grand Challenge when Sebastian’s team won about trying to find a way to work together.

Q. How did the new Google car combine the technology developed for  Stanley, Boss and Pribot? Did you use any one of those projects as a base to build upon? 

C) We took ideas from all of them - different concepts, sensor fusion, map building, etc. The architectures we kind of independently conceived of. Motion planning, we took the concepts and then started here.

Q. So the code was written from scratch? 

All three) Mostly from scratch.

 C) The part you think as kind of the “self-drivingness”, the algorithmic parts of it, that’s all new.

S) There is a reason for that. That is, at least speaking for Stanford, the code was mostly written by graduate students and that moving to the next level you need a much more reliable code base. Looking backwards, the technical achievements necessary to get to the point where we are today, I’d say the Urban Challenge was maybe 20% of where we are today. The vast majority of innovation actually occurred in the recent 18 months.

A) We use many of the same components. And I think there are 2 ways to look at it. One is movements and the capability of the car. But then there is also the quality component. We really focused on that because when you think where we were at the Urban Challenge. We were proving it could be done. Now what we’re trying to do is that it can be done reliably. So we had to do a lot more, not just improvements on the technical side but a lot more improvement on the reliability as well. 

C) Quantifying it, during the Urban Challenge at Carnegie Mellon we awfully did a lot of testing and that was 3,000 miles. We’ve done 140,000 miles of testing at Google. 

Q. And all this code is owned by Google?

C) Yes. 

S) Certainly the work that took place at Google for the self-driving car is owned by Google. But again, there is a ton of software involved that was never written for a self-driving car. 

C) We use open source products too.

 

(Photo: Only one desktop computer in the back)

Q. About the 140,000 miles. How many cars did you have? 

C) We had a total of 8 Priuses to begin with. We had one of them that was rear-ended by another driver so that one is no longer in the fleet. Now we have 7 Priuses and then we are also experimenting with another platform which is the Audi GT which we haven’t used extensively yet, but we’re working on it. 

Q. Why did you decide to use the Prius?

A) I think it’s a question of time. It’s a great robot platform. It was the car easiest to make reliable from a car standpoint. If you look at the reliability, what kind of problems they had at the Urban Challenge, in some cases it was difficult for the cars to actually steer the way they wanted to, slow down and speed up the way you wanted them to. We could use the Prius and make that solid. 

S) Also it symbolizes environmental friendliness because it was the first mass adopted hybrid vehicle. It also has a number of technical provisions inside that makes it easier to turn into a self-driving car. I will give you one example. There are different ways to power steering, one is electric and the other is hydraulic. The Prius has an electric steering which means that you can actually give it an electric signal to the steering wheel which is easier to interface. 

 
(Photo: The equipment beneath the cargo area)

Q. What are the things that the Google car can do now that were not possible at the Urban Challenge? 

C) There’s a lot. Drive at freeway speed. None of the vehicles we had at the Urban Challenge could do that. We can now perceive pedestrians and deal with them. We can merge with freeway traffic. None of the Urban Challenge vehicles you would want to see driving on a public road with traffic. They were very cautious, they were kinda jerky. With our vehicle, you can operate around other traffic. Even people who are videotaping as they go by think that there’s a human driving. And that’s a quality that we didn’t have before. 

Q. To tell whether it’s a pedestrian or a tree, are you using the camera data? 

C) We are not actually using the camera data. Trees don’t move. And it’s not just that. There’s a fair bit that goes into that and we probably don’t want to go into the detail. We can use the laser, we can use our model of the world to detect and track pedestrians. 

Q. To what extent are you using the data from the camera? 

C) For pedestrian tracking, we are not. 

S) We are using it for two purposes. One is for traffic lights, which is another thing we couldn’t do for the Urban Challenge. And for the perception that finds other traffic on highways. The perception comes from multiple layers. Of course some comes from GPS where you are, some level there’s a suite of sensors that are dedicated to find what’s going on around you and those come in different resolutions and different ranges. For the near range, meaning the pedestrians, it’s usually the laser which is most accurate. And we use the radar/camera system in places where the laser can’t reach far enough and it’s specifically the highway where cars need to see far out. 

Q. How many times does the human driver have to go on the route before having the car go driverless?  

C) Once is enough. 

S) We go multiple times just to check. 

Q. And where is this massive amount of being data stored? 

C) As we are driving, we log the altimetry of the car and then we bring it back and upload it to our data repository and then we can extract from that important parts and then we push that back into the car. 

A) It’s just like your navigation system. You have a map in your car. How you got that map is the complicated process. 

Q. But that’s a lot of information. I mean things like construction sites, isn’t that tremendous? 

S) Yes it is. There are specific points where the technology is not yet ready. I will give three examples. One would be construction zones. It’s on our roadmap. The car even today is very safe but it might just ask the person to take over manually in such situations. Second one is double parking. It could cause our car to wait forever, so to speak. Third that we know is snow. We have done no testing in snow. We all believe that we can overcome (these three) in the next months.

It’s important to say that we did not solve the problem of self-driving cars. We made solid progress to this goal. And we are very excited because the problems that we did solve along the way were considered to be very very hard. And we documented this by setting ourselves this 1,000 miles challenge. We set aside from the very beginning a reference set of courses that measured 1,000 miles in total across California which would expose our vehicle to the whole breadth of different types of  challenges. We felt that if there were any fundamental flaws in what we were doing, we wouldn’t be able to accomplish those 1000 miles.

C) Particularly the challenging miles, when you look at some of the historic work, a lot of it is freeway miles. It’s controlled access, very homogeneous, very similar. If you’ve driven 100 miles of freeway, you’ve driven lots of freeway. Whereas if you try to go through Lombard Street in San Francisco or downtown LA or Monterey, these are dissimilar types of roads with all kind of interesting (aspects) which a human driver will take for granted but for autonomous vehicles a lot of difficulty. 

Q. What is the advantage of working on this project at Google? Of course there is the money….

S) One of the things that is happening and that is not quite obvious to the bystander is, that these cars are moving into information vehicles. They massively use data, acquire data, churn data, process data, and use data in every aspect of the driving decision. And that’s exactly Google’s core expertise. For example, massive data that you collect to map our public roadways are being processed by Google’s data center. Normally you don’t think of it this way because classically automotive is about mechanical design. We are taking a strong computer focus. 

C) I think the other big part of it is cultural. Google really has this notion of tackling large and important problems, and that’s one of the things that drew me to here. Organizing the world’s information is kind of a non-trivial problem. In the same way as you would look at self-driving cars. It’s a big space with a lot of opportunity but also a lot of challenges to realize and seeing a company see this as an important problem to solve is inspiring to me. 

Q. How are the car companies feeling about this? 

S) We can’t tell. You would have to ask the car companies, We are certainly talking to car companies about this. For the last couple of years, certainly there are a lot of activities following this direction. There is a huge number of driver assistance systems that perform similar functions to ones that we are using in a self-driving car. 

C) And if you look historically with the DARPA Challenges, the automotive companies were involved and at that time they expressed a shared vision for this technology. 

Q. Are you two going back to university? 

S) I can’t talk about the future. 

C) Same with me. Certainly going on leave, I left with the intent to return.  

Q. Is this project at Google going to continue in the foreseeable future? 

S) This is really more like a science project right now. And we are entering a phase where we are asking ourselves the question of what is the best path moving forward. It is taking place right now. It is hard to say it’s going to go forward for the foreseeable future. There is certainly excitement about the technology at large. We are getting enormous interest from the public constituencies about this technology. But it’s a long way between this and our ultimate goal of using self-driving cars on a day to day basis. 

Q. What is the ideal way that the public can benefit from your research? 

S) The end goal is pretty clear. Today we have close to 40,000 people in the U.S. that are involved in traffic accidents each year. More than a million people worldwide. The American commuter spends 52 minutes in traffic so we would like to make this safer, more fun and productive. We can envision that our technology will assist people with disabilities, people who can’t be mobile and people who lose ability to drive due to aging. So that is the vision.

 And I’m going to punt on the question on how to get there. The reason is we are still in the middle of trying to figure this out. This is more like a science project. It’s too early. While I can see a number of speculative paths, we are still trying to get together and sort it out. We are talking to various constituencies, to companies and so on to try to figure out what is the good path. This is a radically new technology called disruptive technology and it really changes the fundamental assumptions of public traffic. It’s not just an add-on.