Autonomy: The Quest to Build the Driverless Car - And How It Will Reshape Our World

That said, Thrun had learned a lot from his experiences working for Whittaker. In September of 2004, fresh off the heels of Montemerlo’s presentation, Thrun used Whittaker’s template to begin work on his own entry in the second DARPA Grand Challenge. Just as Whittaker did, Thrun recruited volunteers by asking them to enroll in a university class. Thrun’s was called “Projects in Artificial Intelligence.” At the first meeting of maybe forty students Thrun gave a Red Whittaker–style inspirational speech. “Look, there’s no syllabus, no course outline, no lectures,” Thrun recalls saying. “All we’re going to do is build a robot. A robot car that can drive on the original course.”

Thinking of the way Whittaker motivated his students to work hard by providing them with challenges, Thrun set his class a clear and well-defined objective: By the end of the two-month-long session, they were to have built a car that could travel a single mile of the first DARPA Grand Challenge course. “Red and I have very different personalities,” Thrun says. “But I tried to learn from him. And what I learned from Red was, when you give students a goal, no matter how hard it is, because they haven’t learned that these goals are hard to reach, these students think they can reach it. And eventually, they do reach the goal.”

The class didn’t have a budget to go out and buy a car. Someone contacted Ford to ask the manufacturer to donate one, and the company said yes, but they wanted it back afterward, in the same condition they lent it out. Perhaps thinking of Urmson’s rollover accident, Thrun declined the Ford offer. Luckily, a friend of his named Joseph O’Sullivan, an AI researcher who worked for Google, played soccer with a guy, Cedric Dupont, who worked as an engineer at Volkswagen’s lab in Palo Alto. Dupont arranged to provide Thrun’s team with a 2004 Touareg R5 TDI, as well as the help of VW engineers to access its computer system. “That was like a gift from God,” Thrun says. Like H1ghlander, the Touareg had a drive-by-wire interface, and with VW’s help, Thrun’s team could hack into the computer system relatively easily.

Thrun ended up with about twenty people committed to joining the Stanford team, which he split into smaller units. One group was charged with configuring hardware—actually attaching the sensors to the Toureg, which, in a nod to their school, they gave the nickname Stanley. Another part of the team was in charge of providing the mapping. A third handled navigation.

Two months later, at the end of the term, Thrun took his students out to the Mojave Desert and set up Stanley on the course of the first Grand Challenge. Then they activated the robot and watched: Stanley drove past the class’s one-mile goal, thrilling Thrun, who became even more excited when Stanley passed 7.3 miles, which was how far Carnegie Mellon’s Sandstorm had made it. Some minutes later, at 8.4 miles, Stanley found itself stuck in a deep rut, caused by heavy rain.

Thrun was beside himself. The sort of rut that had stymied his robot would have been smoothed over by DARPA prior to the race. Had this been an official race day, it’s possible Stanley would have proceeded much farther. “That was just unbelievable,” Thrun recalls. “That was the moment it became clear to me, boy, there’s a real possibility it can be done.” If a team of comparative novices could surpass the best Carnegie Mellon team in just two months, Thrun wondered, then what could the same team do in the year leading up to the second race?

Red Team’s strategy this time around amounted to a bigger and better version of the approach they’d intended to execute in the first race.

Truth be told, they felt a little cheated by the way the first race went. The communication out of DARPA had led the team to believe that the robots would have to navigate rough territory and brutal off-road conditions. DARPA’s actual route turned out to have some hairy spots, such as tunnels and narrow fence gates. But there was nothing arduous about the road itself. That had been a smoothly graded desert thoroughfare. Your typical subcompact import could have driven off a car lot and navigated it. Looking back, Red Team had wasted countless hours ensuring their robot would be able to handle off-road conditions. And not just handle them—handle them fast. That’s why they’d used shocks and springs to suspend the electronics box and the gimbal, to ensure the computer equipment would be able to withstand the resulting jars and vibrations. Had Red Team forgotten about testing the robot in the most difficult of conditions, and just concentrated on developing a vehicle that would be able to roll from one GPS waypoint to another, then many team members figured they would have ended up finishing the first race. They could have won.

So this time, Whittaker concentrated on refining the capabilities Red Team had already developed, including the pre-driving approach that it had used in the first challenge. In August 2005, Whittaker moved both Sandstorm and H1ghlander out to Nevada. The robotics engineer figured the federal agency would amp up the difficulty for the second race. Some of the toughest roads in the nation were the M1 Abrams tank courses at the Nevada Automotive Test Center. So that’s where Red Team landed with just three months to go, to put the robots, and the team, through a series of what were in effect dress rehearsals designed to replicate race-day conditions—right down to special costumes worn by DARPA staff stand-ins.

Red Team tended to use two different routes to test its vehicles. One, known as the “Pork Chop,” was a 48-kilometer loop that featured everything from dirt road and pavement to cattle guards, high-voltage power lines and railroad crossings. The Hooten Wells route was an 85-kilometer one-way line that followed the course of the Pony Express and featured a dry lake bed, gravel road and a narrow canyon.

The testing featured its share of disasters. Spiker had a credit card linked to a Carnegie Mellon account and was authorized to spend $100,000 a month, a figure he regularly blew past procuring the spare parts required to repair Sandstorm and H1ghlander after the damage caused on their testing runs. For example, on August 26, just twelve days after they arrived in Nevada, H1ghlander sheered off its front right wheel as it navigated a particularly treacherous off-road trail. On September 15, Sandstorm was clotheslined by a tree, sustaining significant, but nevertheless repairable, damage.

These setbacks aside, the testing was going well.

For the first time, Sandstorm and H1ghlander were completing challenge-length runs that featured some of the toughest terrain the team could throw at the robots. The vehicles drove more than 1,600 kilometers each. Better yet, they were completing these runs in times that would have them finishing the race in under seven hours. Red Team was feeling very good about its chances.

Even so, Whittaker was working his team as hard as he ever had. The 4:00 A.M. wake-ups were taking their toll. The race rehearsals started at 6:30 A.M., just like they would during the actual event, and then, after the course work, the team would take the robots back to their garages, where the coders and the mechanics would work long into the night to make improvements and repairs. The next day, they’d rise at four and do it all over again. It was a grueling routine. “Everyone was scraped raw by exhaustion,” Whittaker recalled.

To refresh everyone, to ensure his team was sharp and fully rested come race day, Red set a week’s vacation before the national qualifying event, which began September 28, 2005, at the California Speedway. There, forty-three teams would be evaluated by DARPA, competing to become one of twenty-three finalists to qualify for the actual race on October 8, 2005.

The final day of testing was September 19. Whittaker’s culminating goal had Sandstorm and H1ghlander navigating 10 laps of a 30-mile-long course, to accumulate 300 miles in total, about double what the robots would have to do on race day. Once they achieved the distance, the team would freeze the software, store the robots and disperse to their own chosen habitats for the pre-race rest.

By the afternoon of the nineteenth, Sandstorm was ready for the race but for a last-minute tire and oil change. Meanwhile, H1ghlander was nearing the final laps of its last test session. Following behind in AM General’s second donated Humvee was Peterson in the passenger seat and software engineer Jason Ziglar behind the wheel. Ziglar was doing his darnedest to keep up with H1ghlander, whipping the steering wheel this way and that, his foot jammed on the accelerator. With H1ghlander about to start its final lap, having already gone 270 miles, Peterson called Red in Pittsburgh, where he was handling some last-minute details. “The vehicle is driving really well,” Peterson told him. “But we’re really beating up on it.” What if something happened? Peterson recommended to Red that they call off the final lap. “It felt like we’d learned everything we were going to learn,” Peterson recalls.

Giving up before the team had completed a goal wasn’t in Whittaker’s DNA. He made the call—finish the route. So they kept going. Moments later, H1ghlander was kicking up its usual dust cloud. From the passenger seat in the chase vehicle, Peterson couldn’t see the robot, but thanks to his laptop’s Wi-Fi connection, he could see what H1ghlander could see on the monitor. Approaching a leftward curve, the robot slowed down, the way its algorithms specified, and then accelerated into the curve. Except it swayed just a little bit to the right, off the path—and Peterson’s whole display went red. When the dust cleared, Peterson saw a dirt formation on the right side of the road that looked like the sort of thing a stunt driver would use to shoot a car up into a two-wheeled drive. In this case, the stunt jump had sent H1ghlander over on its side, and ultimately, onto its roof. The robot had caught the right wheels on the ramp at 30 mph and launched itself into the air.

Another rollover.

Having been through this before, the team leapt into action. No one broke down in tears over this one—Spiker was prepared. Many of the extra parts required to repair H1ghlander lay in the mechanics shed at the Nevada Automotive Test Center base. The rest, Spiker arranged to have shipped from Pittsburgh to Nevada.

And that week’s worth of vacation everybody was supposed to go on the next day? Gone. Instead it turned into the biggest work session the Red Team had ever faced.

Once Stanford’s AI class conducted its 8.4-mile test run, Thrun winnowed his team down to four key people. Thrun himself and Carnegie Mellon alum Mike Montemerlo were the first two. Among those who had taken his robot class, Thrun discovered a fellow German, a computer-vision expert and programming whiz named Hendrik Dahlkamp. The fourth was a grad student named David Stavens.

A quartet was appropriate to the task because that’s how many occupants the Touareg comfortably fit. For a week at a time, Thrun and the other three would head out into the Mojave Desert and drive the trails. At first, they’d set the vehicle on a trail, watch it navigate itself, and eventually the robot would encounter something it couldn’t handle. Then someone would code a fix. As the process repeated itself dozens, and eventually hundreds, of times, the robot became sophisticated enough that it began to teach itself. In this phase, Thrun would drive Stanley through the desert, manning the controls, slowing down when the road became rough or steep, accelerating on smooth straightaways. After several days of this, Thrun would go back to the university, and Stanley, working overnight, would retroactively look at the data to engage in its own learning. Confronted with this terrain, Stanley would think, Sebastian chose to drive here—and I will do the same. “The robot would basically spend the night sorting through the data and bring order from chaos,” Thrun said.

Stavens’s contribution was an algorithm that taught the robot how to regulate its speed. The roads Stanley drove in the Mojave featured rain ruts, puddles and potholes. Blasting through this sort of terrain at speed would have shaken the car to pieces. So Stavens wrote a program that regulated Stanley’s progress based on vibrations felt by the robot’s sensors, as well as the grade and width of the road. With the program loaded into the robot, Mike Montemerlo drove Stanley to create data the program could then analyze to develop rules that would guide its behavior.

The problem here was that Montemerlo was too conservative. He’s incredibly detail oriented. A nice way of putting it is risk-averse. “We used to put stickers on his windows,” Thrun recalls. “So Mike couldn’t see how fast we were going.” Montemerlo had once protested to the team members that he would never get in a self-driving car that went more than 5 mph. Driving Stanley, Montemerlo would creep around the desert, easing up hills, wandering over rubble and stones. Then, once the vehicle was at home, the machine learning algorithm would look at the way Montemerlo drove and create rules that would guide Stanley in the future. Accustomed to high-speed driving on Germany’s Autobahn, Thrun didn’t like how slowly Stanley progressed once it had crunched Montemerlo’s data. So one week, when Montemerlo went away on vacation, Thrun set Stanley to go 20 percent faster.

Then came the day in 2005 when Thrun received an unexpected visitor at his Stanford office. He looked up and saw a figure in the doorway. The figure came forward and introduced himself: “Hi,” the man said. “I’m Larry Page.”

Thrun knew who Page was, of course. What surprised him was how interested Page was in the project. “Larry’s always been a robotics enthusiast,” Thrun says, explaining that had Page not started Google, he might have pursued a PhD in robotics. Page was fascinated with Thrun’s project. He had about a million questions. He wanted to see how real the technology was—how close are driverless cars? A century? Decades? A couple of years? What did Thrun think? In fact, Page was so interested that he told Thrun he planned to attend the second Grand Challenge. Through their shared enthusiasm for driverless cars, Thrun and Page developed a friendship that deepened, because the two men both relished taking on tasks that everyone else dismissed as impossible. Thrun had no idea, at that point, that Page would change the course of his life.

At 4:30 A.M. on October 8, 2005, the day of the race, DARPA officials provided a Red Team member a USB key featuring a computer file of 2,935 waypoints—the course of the second Grand Challenge. The whole of the route totaled 132 miles, starting and ending in Primm, Nevada.

The next bit bore many similarities to the first race. The team member sprinted to Red Team’s command center. Another member loaded the route network definition file onto Red Team’s shared hard drive. A computer program analyzed the waypoints and added thousands more, so a route originally specified every eighty yards now featured a dot every yard or two. Next, the route was divided up among team members to go over. The pre-planning team went through each part of the route to ensure the new waypoints kept Sandstorm and H1ghlander on navigable road.

In the anxious moments that passed while the pre-planning team worked, Whittaker, Urmson and Peterson discussed strategy. The experience of the first race eighteen months before was fresh in everyone’s minds. That time, they’d gone for speed. And perhaps they’d pushed Sandstorm beyond what was good for it.

So the three decided Red Team should take a tortoise-and-hare approach with its two vehicles. One of the vehicles would take it easy, going so slowly that it would be certain to finish the race. This way, in the event that no one else finished, at least Red Team would have a vehicle that crossed the finish line.

Sandstorm consistently came in 10 percent slower than H1ghlander—a symptom, the engineers thought, of the way the electronics box floated, which made it difficult for the robot to pinpoint exactly where it was. So H1ghlander would be Red Team’s hare, while Sandstorm was the tortoise.

In terrain the pre-planning team classified as moderately difficult, H1ghlander would go 20 percent faster than Sandstorm. In very safe territory, Whittaker decided that Sandstorm was allowed to go 27 mph, while H1ghlander was able to go up to 30 mph—an increase in speed of 12.5 percent. H1ghlander, Red said, should target to finish in 6 hours and 19 minutes, for an average speed of 21 mph. And their safety, Sandstorm, should finish in 7 hours, 1 minute.

Urmson and other Red Team members watched the race from Stanford’s tent, because Stanford had the best view. H1ghlander was first out of the starting chute. And in the initial few miles, it led the pack. Then, nearly seventeen miles in, H1ghlander faltered. The engine stalled and the vehicle came to a stop, then started again. Coming up on a hill, it stalled again. This time, the robot actually rolled backward. It crested the hill on a subsequent attempt, but still, nothing like this sort of engine trouble had ever happened in testing.

Red Team had people stationed at designated viewing points that DARPA had set along the course. Reports came back that another engine stall likely happened fifty-four miles into the race. The stalls prevented the engine from turning a generator that created electricity for the sensors. Backup batteries were able to provide some power, but not enough for the main LIDAR unit. That was set in a gimbal, which a helicopter camera crew revealed was positioned at a ninety-degree angle to the direction of the robot’s travel, rendering it completely ineffective.

The disabled robot slowed so much that the second entry to leave the chute, Stanford’s Stanley, caught up to H1ghlander at mile 73.5. DARPA had promised its contestants that their robots would be navigating a static environment, meaning nothing could move in any of the contestants’ fields of view. To prevent Stanley and H1ghlander from confusing each other, DARPA used a radio transmitter to “pause” Stanley for 2 minutes and 45 seconds, allowing H1ghlander to go ahead, creating some territory between the two robots. But soon after Stanley was reactivated, the robot caught up to H1ghlander a second time. This time DARPA paused Stanley for 6 minutes and 35 seconds. But Stanley caught up to H1ghlander a third time. Finally, at mile 101.5, 5 hours, 24 minutes and 45 seconds into the race, DARPA paused H1ghlander and allowed Stanley to take the lead. “Stanley has passed H1ghlander,” Tether announced in the observation tent, prompting Thrun to leap into the air in triumph.

Shortly after, and with an elapsed time of 6 hours, 53 minutes and 58 seconds, Stanley became the first robot ever to autonomously complete a DARPA Grand Challenge. Tether himself waved the checkered flag as Stanley passed the finish line.

Sandstorm launched at about 6:50 A.M. The robot rumbled out of its chute with its characteristic diesel knock. It made it through underpass one, two and three even though a software bug prevented the LIDAR from detecting the walls. In fact, it performed flawlessly until 6 hours and 30 minutes into the race, when it just scraped a canyon wall in the narrowest section of the route. Sandstorm drove over the finish line 7 hours and 4 minutes after it left the start chute—a variation of only about 1 percent from what the engineers had asked of the robot. It had done exactly what it was assigned to do in remarkable fashion, placing second, by time. And in third, limping into the finish, was H1ghlander, with an elapsed time of 7 hours and 14 minutes, or 55 minutes longer than the time the Red Team had set for it. All told, five robots finished the course.

Thrun was elated, of course. Later that day he and his team gathered onstage to receive a check for $2 million. But what was just as gratifying was the way the victory felt like a validation of the whole robotics field. More than a decade later, public attitudes toward roboticists have markedly changed. Back in 2005, robotics was associated in the public imagination with projects like Thrun’s 1998 Minerva museum tour guide—as novelties, curiosities that had little effect on anyone’s day-to-day lives. A self-driving car was different. Sure, the second DARPA Grand Challenge was a controlled scenario separate from the actual world because nothing else on the course was allowed to move. But it nevertheless represented a step toward actual robot cars, which everyone realized would, if they ever became a reality, transform lives. Standing up before reporters frantically scribbling down their words, photographers and videographers capturing their images and a crowd of people cheering their accomplishment, Thrun and his teammates relished the attention as validation that the world might finally recognize the potential of their chosen field.

Thrun was magnanimous in his victory. “It’s really been us as a field that were able to develop these five vehicles that finished the race,” Thrun said. “It’s really been a victory for all of us.”

Few on Red Team felt that way. It stung that they had devoted months to test Sandstorm and H1ghlander on some of the toughest roads on the planet—and then discovered on race day that the course was easier even than the well-graded roads that had marked the first Grand Challenge. It stung, too, that based on its performance in the qualifying events, a fully functioning H1ghlander would have taken the race. And it stung that, had Red Team’s leadership allowed Sandstorm to perform to its abilities, rather than playing it safe and limiting its speed, the older Red Team robot also might have beaten Stanley. Thrun acknowledged both facts. “It was a complete act of randomness that Stanley actually won,” he said later. “It was really a failure of Carnegie Mellon’s engine that made us win, no more and no less than that.”

“It was very much a winner-take-all event,” Urmson recalls, more than a decade later. “It sucked. There was no prize for second. This had been three years of people’s lives at this point. It was brutal. I remember seeing Red afterward, and that was the most distraught I’d ever seen him.”

“It’s right up there with the worst shortcomings of one’s life,” Red says, assuming full responsibility for what he still regards as a defeat. “I let a team down. I let a lot of people down. And in a lot of ways, in a bigger way, I let down a community and a world that didn’t see the best of the technology and the movement and the vision of what things could be.”

“It was a strange feeling,” Urmson says. “It was a day that five vehicles did something believed to be impossible. Our team had pulled together and achieved the impossible. We’d done the impossible—and yet we’d lost.”

Chapter Three (#ulink_6fb98a20-dd46-5871-9dc7-e851df6392f0)

HISTORY HAPPENS IN VICTORVILLE (#ulink_6fb98a20-dd46-5871-9dc7-e851df6392f0)

An introverted engineer looks at his shoes when he talks to you. An extroverted engineer looks at your shoes.

—UNKNOWN

The second DARPA Grand Challenge was successful on numerous fronts. The $2 million prize was perceived as a cost-effective way to spur progress in the field of mobile robotics. The large number of entrants, the public enthusiasm and media attention for the challenge event, and the fact that the race resulted in five vehicles that could travel 132 miles through a difficult desert landscape all contributed to a perception throughout the military of money well spent.

But inside DARPA, there remained a sense that the mission hadn’t yet been accomplished. No team had constructed a robot that could navigate the chaotic urban environments of Iraq or Afghanistan. Could a similar event spur the robotics field to make more progress?