Autonomy: The Quest to Build the Driverless Car - And How It Will Reshape Our World

Because Sandstorm had performed best in the qualifiers, it had the honor of starting first. The big Humvee rolled slowly out of its chute. “Ladies and genetlemen, Sandstorm!” Tether cried. “[An] autonomous vehicle traversing the desert with the goal of keeping our young military personnel out of harm’s way.”

The first complication in the race course was a leftward turn. Its inside edge was marked with some scrubby vegetation, and its outside edge, with concrete jersey barriers protecting spectators from the robots. Sandstorm followed the road perfectly throughout the curve and accelerated once it headed out on its straightaway.

While it was still in view, Sandstorm ran over a hay bale. Urmson winced. But the big off-road vehicle just kept on going. Soon, the Red Team couldn’t see their robot at all. No one had thought to provide the teams with a video feed of their vehicle’s progress. All they could do was settle in and wait to hear reports issued back to the start from helicopter-borne observers and other officials set along the course.

Soon, the other entries headed out: A team called SciAutonics II. Then CajunBot rolled its six wheels from the starting chute and drove straight into a jersey barrier. Team ENSCO’s robot, based on a Honda ATV, wandered from the road just past the turn, flipped over on its side and was out of the race just two hundred yards into the event.

Palos Verdes High School’s autonomous SUV also ran into a jersey barrier. And then came the most curious of the entries: Anthony Levandowski’s autonomous motorcycle. Levandowski pulled it up to the starting line, activated the gas-powered motor, stepped away—and watched, brokenhearted, as the motorcycle immediately tipped over. As Levandowski would discover later, he’d forgotten to activate the gyroscope that kept the motorbike balanced. His race was over.

Minutes later, Red Team heard from a race organizer that something was wrong with Sandstorm. The hay bale the robot had run over just after the start turned out to reflect an ongoing problem. Perhaps because its sensors hadn’t been calibrated properly, perhaps because the main LIDAR’s replacement unit scanned at a much slower rate than the original, Sandstorm consistently appeared to think that it was a foot or two to the left or right of where it actually was. The Humvee drove over a fencepost, then another and a third. Some miles later, the vehicle swung itself into a curve, a particularly tricky one given the inside edge was separated from a steep drop-off by only a knee-high berm. As Urmson and Peterson intended, Sandstorm slowed down as the road turned. But the robot was a foot or two to the left from where it should have been. As a result, the left-most tires climbed up the berm, then dropped down the steep inside ledge. Sandstorm was now stuck on its belly—what Urmson called “high-centering.”

Things quickly grew worse. Sensing Sandstorm wasn’t moving, the speed control system kicked in, directing more power to the engine. One of the tires hanging over the other side of the berm was situated just high enough off the ground that it could still touch the Mojave sand. The friction heated up the rubber until it smoked and eventually burst into flames. The robot’s progress was over 7.3 miles from its start.

The media used Sandstorm’s flame-out as a metaphor for the entire event. The number-two entrant, SciAutonics II, also got stuck on a low hill of earth. Dave Hall’s Toyota Tundra became confused by a small rock. The UCLA entry, Golem Group, stalled out when a safety device prevented its engine from accelerating enough to get up an incline. And TerraMax, the 32,000-pound monster truck known for its brute force approach, ended up halted when a pair of tumbleweeds it incorrectly considered immovable obstacles blew ahead and behind it. And those were the best-performing vehicles.

The result put DARPA director Tony Tether in a tough spot. At the other end of the race course, in Primm, Nevada, was a tent full of reporters who had traveled across the country to file stories on the race winner. Tether figured he was going to get killed by the press—an expectation that proved right. “DARPA’s Debacle in the Desert,” went one headline. The gist of the stories portrayed DARPA as an out-of-touch government bureaucracy that had wasted money staging a fool’s errand. So to distract them, Tether took the stage and announced a second race, to be held in a year or so, with a doubling of the 2004 race’s purse, to $2 million.

Chapter Two (#ulink_95b72bad-ce1f-56ed-b430-24775e59f3a4)

A SECOND CHANCE (#ulink_95b72bad-ce1f-56ed-b430-24775e59f3a4)

The only way to prove you’re a good sport is to lose.

—ERNIE BANKS

Red Whittaker started planning for the second race even before Sandstorm returned to Pittsburgh from the first. Through his repeated entreaties for sponsorship, Whittaker had developed a relationship with AM General, the company that manufactured the Humvee. Now Whittaker thought he could convince the executives to donate an additional vehicle for Red Team to use in the next challenge—if the executive team would only witness a demonstration of Sandstorm’s capabilities.

Several days after the first challenge, Whittaker, Spiker and Peterson arrived with Sandstorm at the AM General campus in South Bend, Indiana, to conduct that demonstration. Spiker and Peterson stayed outside and set up the robot on an obstacle course the Humvee manufacturer maintained to educate new owners on the capabilities of their vehicles.

One element of the obstacle course was a concrete tabletop structure, maybe eighteen inches off the ground. Peterson and Spiker wondered whether Sandstorm could drive itself up and onto the obstacle. Moments later, rather than creeping toward the tabletop, as Spiker and Peterson had intended, Sandstorm took off toward it at high speed.

A kill switch was designed to deactivate Sandstorm if it ever did anything unpredictable. Trouble was, the kill switch had about a two-second delay. Spiker pressed the switch, but Sandstorm hit the tabletop before the command took effect. The front wheels bounced the front end into the air. The rear wheels hit the tabletop and bounced up the Humvee’s back end. For a moment the entire vehicle was airborne. Then the front end nose-dived with a violent slam against the concrete.

That’s when the kill switch disabled the vehicle.

Spiker and Peterson rushed to assess the damage. Whittaker was in a nearby building conducting his presentation for AM General executives on Red Team, and the wonderful capabilities of the robot they’d developed. Outside, Spiker and Peterson discovered the impact of Sandstorm on the tabletop had crushed an engine-compartment coolant tank. Once that was repaired, they set up Sandstorm on a section of clear road and activated the giant robot to test it. Immediately the front wheels turned to the right. That shouldn’t have happened. “Kill kill kill!” Spiker shouted to Peterson. With a snort of exhaust, Sandstorm accelerated right off the road and straight into the building where Whittaker was talking to the AM General executives. The impact of the Humvee against the wall shook the entire structure.

Later, Spiker figured out that the tabletop collision had detached a steering position sensor from its mooring—which, in turn, caused the second accident. But it turned out not to have mattered. Whittaker and the AM General executives rushed from the building to investigate the source of the impact. Spiker figured the sponsorship bid was toast. But as the execs surveyed the scene of the accident, Spiker realized his fears were groundless.

“Unflinching grace” is the way Whittaker characterizes the AM General execs’ reactions, portraying them as “great hosts who don’t fuss over a dropped fork or spilled water.” The executives saw themselves as manufacturing a vehicle designed to push the bounds of what an automobile could do—and so, in its own way, did the Red Team. Of course they would sponsor Whittaker’s team. “We’ll give you two Humvees,” one of the AM General execs proclaimed. “Just be careful.”

Some months later, in the summer of 2004, a computer scientist named Sebastian Thrun listened to a presentation about the first DARPA Grand Challenge in a seminar room at Stanford University. Thrun had recently moved from a faculty position at Carnegie Mellon’s Robotics Institute, where he’d been working on a project with Red Whittaker—a robot called Groundhog that was designed to map Pennsylvania’s abandoned coal mines. His new job was in Palo Alto, California, leading the Stanford Artificial Intelligence Laboratory, a once-respected research facility established by AI pioneer John McCarthy in 1963, which had been dormant since it had been rolled into the greater computer science faculty in 1980. To reincarnate the facility, Thrun brought nine Carnegie Mellon academics with him. Having left behind all his projects at his old school, Thrun was looking for a quick way to reestablish the AI lab’s reputation.

Thrun had attended the first Grand Challenge as a spectator, and was intrigued by the prospect of entering the second, as the rebooted Stanford AI lab’s first major feat. So Thrun asked one of his fellow CMU transplants, who had also attended the first challenge, to conduct a presentation to the rest of the group.

The presenter was Mike Montemerlo, a soft-spoken engineer who had a reputation as a software whiz known for his ability to program robots to conduct the simultaneous localization and mapping that had so bedeviled Sandstorm in the first race. Montemerlo’s father, Melvin Montemerlo, was a program executive at NASA and had worked closely with Whittaker on numerous projects. When Mike had been in high school, his dad had taken him on a pre-college trip to experience firsthand candidate campuses. One evening in Pittsburgh, the pair of them threw pebbles up at Whittaker’s window to convince the robotics legend to give the teenager a tour of the Field Robotics Center. That experience was the reason Montemerlo attended CMU. Years later, Whittaker would become Montemerlo’s PhD adviser; in the same period, Montemerlo also happened to be Chris Urmson’s officemate.

At Stanford, Montemerlo’s presentation amounted to a travelogue of his experiences at the California Speedway. Full of photos of the various robots, the seminar highlighted the problems and foibles that each team experienced. He spent a lot of time on the work that had almost been destroyed by Sandstorm’s rollover accident. The penultimate slide asked whether the Stanford AI lab should compete in the second DARPA Grand Challenge. The final slide featured the answer: “No,” in bold and all caps.

Thrun is a slim man who communicates in perfectly enunciated, precisely formed sentences colored with a German accent; he was born in the small Rhineland city of Solingen and raised in north Germany. “Why not?” he asked softly.

“It’s hard,” said Montemerlo, whose side-parted brown hair and wire-framed circular glasses made him resemble the Hollywood stereotype of a software engineer. “It’s all encompassing,” he followed up, perhaps thinking of the experience of Urmson and the rest of the CMU team. “People have to work all day and all night. They lose their social life. And—it can’t be done!”

Somewhere, somehow, Montemerlo must have known that telling Thrun that something couldn’t be done was the quickest way to entice him to try it. “I’m a rule breaker,” Thrun says, a character trait he shares with Whittaker. “A rebel—I like to do crazy things.”

Thrun was the third of three children. “I was the one the parents didn’t have the energy and time to pay attention to,” he told one reporter, years later. “I remember a beautiful childhood—but pretty much on my own.” Left to his own devices, he developed various obsessions with intellectual projects. At the age of twelve, in 1980, the obsession involved a Texas Instruments pocket calculator that could be programmed to solve various equations. Thrun delighted himself using it to create little video games. Next, he happened upon a Commodore 64 personal computer on display in a local department store. The computer was too expensive for his middle-class family, so Thrun returned to the store display to program on it, day after day, week after week. Each day he tried bigger and bigger programming challenges. He grew adept at efficient coding; because the staff turned off the computer each night, he had to execute each challenge he set himself in the two and a half hours that passed between the end of the school day and the store’s closing time.

By the time Thrun’s parents bought him a used NorthStar Horizon personal computer, the young man was able to program simple video games. He wrote a virtual simulation of the Rubik’s Cube. Another feat involved coding the member database for his family’s tennis club. One gets a sense that Thrun roved through his adolescence seeking out challenging problems that he would use to test his programming ability. The same method would predominate in Thrun’s academic and professional life. He enrolled in the computer science department at the University of Bonn. Artificial intelligence attracted him because, in comparison to humans, with their sometimes irrational, inscrutable behavior, Thrun felt he could fully grasp the reasons a software program acted the way it did.

In 1990, the University of Bonn bought a Japanese robotic arm as a research tool. Thrun distinguished himself by using a neural network to teach the robot how to catch a rolling ball. The resultant academic paper was accepted to an American artificial intelligence conference, Neural Information Processing Systems. The trip was a turning point for Thrun, who was then twenty-two. He’d discovered people exactly like him—a whole community of “psychologists and statisticians and computer scientists all working together to understand how to make machines learn.” From that moment on Thrun focused on writing academic papers so he could attend more AI conferences. Through such gatherings, Carnegie Mellon AI legend Alex Waibel became a mentor, as did Thrun’s future thesis adviser, Tom Mitchell. Thrun joined the CMU faculty after he earned his PhD in computer science and statistics from the University of Bonn in 1995.

One of the most interesting projects Thrun worked on in Pittsburgh was the creation of a robot tour guide for museums. In keeping with the kitsch factor the public associated with robots—think the 1986 comedy Short Circuit, the TV show Knight Rider and Data, the well-meaning android on Star Trek: The Next Generation—the tour guide that Thrun constructed, Minerva, included a pair of camera lenses for eyes and a red mouth that could tilt into a frown to indicate displeasure. As a publicity stunt to demonstrate the capabilities of technology, Minerva even provided tours to visitors of Washington’s Smithsonian Museum.

It turned out programming a robot to navigate through a museum was a surprisingly complex challenge. Minerva would share the museum floor with dozens of human tourists—as well as valuable museum exhibits. How to engineer the creature so that it didn’t bump into an exhibit? How to write the code so that it didn’t roll over a child?

Six years before DARPA staged its first Grand Challenge, in 1998, Thrun equipped Minerva with laser-range finders. Then he loaded the robot with a machine-learning algorithm and sent it out on the museum floor at night, without any tourists around. Minerva wandered around the exhibits, sending out laser beams and creating a map of its environment. Then, when the museum was open, with humans sharing the same floor as the robot, Minerva would use this map to locate itself. The map also provided a way for Minerva to avoid running into humans. The robot would assume any new obstacle that hadn’t been on the original map was a human, causing Minerva to stop safely.

The tour guide was a big hit, and Thrun used the acclaim to handle the software side of other projects. For example, Whittaker convinced Thrun to join the team that built the Groundhog robot that aimed to make it safer for Appalachian coal miners to retrieve their underground ore. Maps didn’t exist for older, decommissioned mines in the area, which could cause problems. In 2002, for example, nine workers toiling in Pennsylvania’s Quecreek mine were trapped by water when they breached an adjacent passageway that had been abandoned for years and flooded sometime along the way. The miners escaped after three days, but Whittaker took the accident as a challenge and, in just two months, with Thrun working on the SLAM programming, created a robot that could be dropped into old mines to scan the passageways and create 3-D maps for reference.

DARPA’s series of challenges fascinated Thrun. When Thrun was eighteen, in 1986, his best friend, Harald, was invited for a ride in another friend’s new Audi Quattro. It was an icy day, and the driver was going too fast and ran the Quattro headfirst into a truck. Harald died instantly. The impact was so strong that his seat belt was shredded. The crash would forever haunt the German robotics professor.

Thrun saw self-driving cars as a way to make automobile transportation safer, to avoid crashes like the one that killed his friend. He did some thinking about the problem after the first Grand Challenge. The fact that DARPA created waypoints along the route really simplified the problem, he figured. Programming Minerva to navigate the fast-changing and crowded environment of the Smithsonian Museum rivaled the complexity of the self-driving-car problem. Before he left Carnegie Mellon, he went to Red Whittaker with an offer. “Look,” Thrun told the older robotics legend. “I’ve been recruited from Stanford, but for the next Grand Challenge, I would love to help you.”

“Had he said yes,” Thrun recalls, “I would have happily served on his team and never have started my own team.”

But Whittaker declined Thrun’s offer, presumably because he wanted to keep Red Team exclusive to people associated with Carnegie Mellon. After Montemerlo’s presentation, Thrun considered whether to enter the second challenge himself. Red Team had taken a year to build a robot that went 7.3 miles. If Thrun’s new lab could do better, they’d go a long way toward establishing a national reputation. SLAM would be integral to a successful performance, and Thrun and Montemerlo were two of the world’s leading experts on the topic. Thrun basically figured, why not?

So on August 14, when DARPA staged a conference for potential competitors, Thrun brought Montemerlo and several other members of his team. The conference was held in Anaheim, California. Despite the negative media coverage of the first race, even more competitors came out this time around: more than 500 people from 42 states and 7 different countries attended the 2004 competitors’ conference. Ultimately, 195 teams would register to compete, nearly double the number that signed up for the first race.

Including, of course, the Red Team. The summer after the debacle in the desert, Urmson went off and completed his PhD, then got a job working for Science Applications International Corporation, the government contractor that had sponsored Sandstorm. Urmson’s assignment was to work with Red Whittaker and Red Team on the second DARPA race. Urmson’s hopes were considerably higher for the second challenge. They’d have another eighteen months to perfect Sandstorm’s development. And they’d be doing so with a more professional group, including several engineers from Caterpillar, the construction-equipment manufacturer. The budget was bigger, at $3 million. The atmosphere was different, too. The first time out there was youthful enthusiasm. This time, there was an almost grim determination.

“I signed up to win the Grand Challenge,” Whittaker proclaimed. “This time around, the Red Team will be more like a Red Army.”

It was inevitable that the Stanford and Carnegie Mellon teams would bump into each other at the preliminary conference. Urmson noticed that Montemerlo was carrying a sheaf of papers in his hand that turned out to be the technical paper Urmson had written after the first race. The paper described the most intimate details of Red Team’s approach. Publishing for the rest of the robotics community the secrets of all competitors’ approaches had been one of DARPA’s conditions of entry. It was a good strategy. In the spirit of academia, sharing intelligence meant the whole field progressed faster. But it also made things more difficult for Whittaker and Urmson. As the country’s leading robotics lab they’d had a head start for the first race. Publishing their approach brought everyone else closer to the Red Team’s level. And the defectors, Montemerlo and Thrun, were brilliant people. That they were entering meant the prize was no longer Carnegie Mellon’s to take. Now, heading into the second challenge, Red Team faced its most serious competition yet.

Early on in its preparations, Red Team decided to hedge its bets by entering two robots. (There was a precedent for this. SciAutonics had entered two vehicles in the first race.) Partially, the step was designed to smooth relations between team software lead Kevin Peterson and project manager Chris Urmson, who were apt to butt heads in the latter half of Sandstorm’s development. There was talk of giving each deputy his own vehicle, although years later Whittaker would insist that Peterson and Urmson contributed to both robots in the lead-up to the second race. And partially, the move was pragmatic. After all, thanks to AM General’s donation, Red Team had enough Humvees.

The second vehicle, which became known as H1ghlander, was a 1999 model year, making it thirteen years younger than Sandstorm. The AM General–donated vehicle came with a 6.5-liter turbocharged engine. One of the challenges of autonomous driving involved controlling acceleration and steering. Most vehicles of the era were mechanically controlled. They relied on a human being twisting steering wheels, pushing accelerators, shifting gears, which complicated matters when a computer was supposed to do the driving. There was a margin of error when a digitally controlled actuator pressed against, say, a gas pedal.

This new Humvee, H1ghlander, featured drive-by-wire capability embedded in its controls. It had been designed to be controlled by a computer. The throttle, for example, was operated by a factory-installed engine control module. So instead of rigging up an electric motor and lever to actually push against the gas pedal, as with Sandstorm, the H1ghlander crew could hack into the newer Humvee’s computer system and control the throttle electronically. It all meant less margin of error, which made H1ghlander a better driver.

Another change was that Whittaker and his students had tracked down a different, more accurate location-tracking system. The system used in the first race had a margin of error of about a yard. This new one, from a sponsor named Applanix, featured a margin of error of about twenty-five centimeters, or less than a foot—a big improvement for the second race.

So the Red Team had a lot going for it. But so, too, did Thrun’s team. In his heart, Whittaker was a hardware guy, who came from an era when making robots work involved the precise interplay between actuators and carburetors, electric motors and solar-powered chargers. This was reflected in Red Team’s approach to the first challenge, which saw his charges spending as much time perfecting the e-box and gimbal mechanisms as writing code for the computers. But as computing power improved, robotics was increasingly becoming a software problem, which computer scientists, rather than mechanical engineers, had to solve. Whittaker was an engineer. Thrun’s team was dominated by computer scientists. Very little of the hardware that Stanford used needed to be custom-designed. In contrast to Sandstorm’s gimbal and e-box, which the Carnegie Mellon team had engineered itself, Thrun simply took sensors he found in the marketplace and bolted them to his team’s vehicle, including five LIDAR units, a color camera to aid road detection and two radar sensors designed to identify large obstacles at long distances. The philosophy of the Stanford team was to “treat autonomous navigation as a software problem.”

“My perspective was, you take a human out of a car, and replace it with a robot—there’s a bit of a hardware issue,” Thrun observes. “You have to figure out how to crank the steering wheel and press the brake. But that part is trivial. You put a little motor on the steering wheel. There’s no science … It’s all about artificial intelligence. About making the right decision. So we had this complete focus on making the system smart.”

“Carnegie Mellon was a team—it’s a humongous place, and they have experts in everything,” Montemerlo explains. “We were a much smaller group. We very much were software people. None of us had any mechanical skill whatsoever.”