Autonomy: The Quest to Build the Driverless Car - And How It Will Reshape Our World

Urmson grew up among people who viewed with suspicion those who drew attention to themselves. What the guy is, is solid. Straight-shooting. Steady. Urmson is not the guy you’re going to notice first when you walk into a room. But you spend enough time with the people in that room, and I don’t care who is in there, after a while Urmson will be the guy you trust to lead—to carry out the plan.

And in April 2003, Chris Urmson had a plan. In fact, Urmson thought he had the next couple of years of his life pretty well figured out as he drove from the remote Chilean city of Iquique to the great salt plains of the Atacama Desert. The road from Iquique into the Atacama would make anyone nervous. It zigs and zags from the Pacific Ocean on up a near-vertical shelf. Those who remember plate tectonics from high school geology might recall that this is where the Pacific’s Nazca Plate collides with South America, pushing the continent into the air, creating a ridge thousands of feet high and a rain shadow that runs six hundred miles up and down the Chilean coast. That rain shadow is the Atacama. One of Earth’s most forbidding landscapes, the driest nonpolar desert on the planet, an area so desolate that scientists use it as a stand-in for Mars. That was what Urmson was doing there. He was one of a handful of roboticists joining a team of NASA staff members to test a robot designed to crawl across the Martian landscape to seek out signs of life.

At twenty-seven, Urmson was tall and athletically built, with sandy blond hair and smiling blue eyes behind round, wire-framed spectacles. He tended to jam a baseball hat so low over his brow that the brim would touch the top of his glasses. Urmson planned to spend about a month in the Atacama. Then he’d return to Pittsburgh, where he was a graduate student in the robotics program at Carnegie Mellon University. He’d write up his dissertation, undergo the grilling every thesis committee is supposed to do, hopefully get his PhD and then a job—maybe join the faculty of his alma mater’s Robotics Institute, home to more robotics brainpower than anywhere on earth, or maybe join one of the start-ups that occasionally spun out of the university. In any event, he’d start making money, enough for him and his wife to have the kids they’d been putting off while Urmson finished his studies.

The campsite that Urmson’s research group chose amounted to little more than a handful of bright yellow dome tents, a slightly bigger meeting tent—where they kept the computers—and a pickup. And Hyperion. Hyperion was the robot. Not the conventional kind of robot. No arms and legs. Rather, Hyperion sat on a quartet of bike tires and was roofed with solar panels and powered with an electric motor. Hyperion was the reason why Urmson, and his fellow scientists from Carnegie Mellon and NASA’s Ames Research Center, had traveled over half the planet.

Hyperion was designed to wander across the Martian surface, sniffing and scraping and testing the soil for signs of life. Urmson was in charge of programming the software that dictated how fast Hyperion rolled.

The scientists took their breakfasts and dinners at a nearby salt mine. Nights, they sat around a fire and watched the camanchacas roll in, the Pacific salt fog that could rust exposed metal within a single night. They turned in to tents they used for warmth rather than protection. You camp in other deserts, you need a tent to keep snakes out of your sleeping bag and scorpions out of your boots. But nothing lived in the Atacama Desert. Not snakes. Not scorpions. The only living things that Hyperion’s minders saw were vultures.

The encounter that would change Urmson’s life started with the sight of a long dust cloud led by a speeding pickup. Some minutes later, the dust cloud followed the pickup into the Hyperion campsite. The door opened, and out of the truck popped William L. Whittaker, commonly referred to as Red.

Whittaker was another big guy, an inch or two taller than Urmson at about six-foot-three, with shoulders that look like they’d brush the sides of interior door openings. His scalp is closely shorn; years ago, when he did have hair, the color of it was what gave him his nickname. His gaze is intelligent and contemplative. It feels like his eyes can see right into your soul when he looks at you. Anyone who spends five minutes with Whittaker can tell that he spent formative years in the U.S. Marine Corps. He speaks in the sort of aphorisms that drill sergeants put on their bedroom walls. “Winning isn’t everything,” he might say. “It’s the only thing.” And: “Worry is a formula for failure.” Another favorite: “If you haven’t done everything, you haven’t done a thing.” Hyperion was somewhere around the sixty-fifth robot Whittaker had worked on in his career as a roboticist.

The Carnegie Mellon professor strode out of the pickup in boots, conducting a round of handshakes with his big hands. He was there in part because he was Urmson’s thesis adviser, and he was checking on his charge. But you could tell that Whittaker was holding something in. Something big. Pretty soon, Whittaker came out with it. The U.S. Department of Defense was staging a driving race for robots. Specifically, the Defense Advanced Research Projects Agency. DARPA, Urmson knew, was the U.S. government’s developmental laboratory, credited with spurring such useful inventions as drone technology and the Internet (a military invention whose distributed knowledge network was intended to safeguard the records of the U.S. government in the event of a nuclear attack). DARPA was also responsible for such less-than-useful innovations as mechanical lobsters for the U.S. Navy and DNA-editing techniques intended to create humans who didn’t need sleep. Now DARPA Director Tony Tether was turning the agency’s direction toward autonomous cars.

For years, Washington had pushed American defense contractors to develop autonomous technology so that a third of all American military vehicles could be self-driving by 2015—a stated mandate from Congress. In the aftermath of 9/11, the effort took on added urgency as the U.S. military lost infantrymen and -women to improvised explosive devices planted under the roads in Afghanistan and Iraq. If self-driving vehicles ever became possible, military robots might drive themselves over the sort of desert roads found in overseas theaters of war. But the four-star generals had been frustrated with the pace of change. The problem was proving too difficult for the military contractors. And so Tether struck upon a novel solution: DARPA would stage a race. For robot cars.

As Whittaker recounted the details to Urmson, they sounded a little insane. DARPA said it would allow any American team to enter—student, hobbyist, professional, whomever. The course would bisect the Mojave Desert, running eastward from Barstow, California, to Primm, Nevada, for a distance of about 150 miles. The prize money would go to the first team that could do it in under ten hours.

“Wow,” Urmson said, thinking Whittaker was just making conversation.

But Whittaker never just made conversation. The prize money, the old marine said, was a million bucks. And Whittaker wanted to win that money with Urmson’s help.

It would be three years before I met Chris Urmson, who would go on to become one of my favorite people. But I can see how this situation would have presented him with a dilemma that contradicted two of his prime directives. Urmson had a seemingly innate desire to try to improve the stupid and inefficient things about the world; he once interrupted an important business meeting at a Pittsburgh coffee shop to burst out onto the street and direct traffic, just to help someone turn left out of a parking lot. He was programmed with an engineer’s duty to seek out the coolest and most interesting projects that could change the lives of the most people. Which is why Hyperion was such a perfect project for him. How could you get cooler than an autonomous robot designed to seek out life on other planets?

Actually, it turned out that you could. Urmson’s work with Hyperion was helping the robot travel anywhere from 15 to 25 centimeters a second—about the pace of a slow walk. In the DARPA race, the robot would have to travel 150 miles in at least 10 hours, which required an average speed of about 15 mph, as fast as most cyclists went. The speed, the money, the fact that the race was intended to address an issue killing American soldiers overseas—Urmson got it. He ached to participate.

But there was a problem: He was also programmed with a duty passed down to him from his parents, to do what was best for his family.

Chris Urmson was born in 1976 to Paul and Susan Urmson, an English couple who had immigrated to Canada because they thought it would represent better opportunities for their three sons. Paul’s first career was as an electrician, and then, once his kids were born, he pursued his college degree at night school, earning his BA and then master’s. Susan enrolled in nursing school after the kids were born and went on to administer methadone programs within the Canadian prison system.

The point? The three Urmson boys grew up in homes where the parents were always working, always bettering themselves for the sake of the family and where education was prized from the kids’ earliest ages. The Urmson parents ran their lives for their children. The family moved a lot because Paul’s work in the prison system required him to transfer around the country. Each time they did, Paul and Susan settled the family in the cheapest house in the nicest neighborhood they could find—a strategy they devised to send their kids to the best public schools. The strategy worked. In addition to birthing one of the most important engineers in the development of autonomous cars, the Urmsons also raised an orthopedic surgeon and a Mountie, a member of the Royal Canadian Mounted Police, which is something of a trifecta for middle-class families north of the border.

At a young age, Chris’s teachers assessed him as gifted, which qualified him to attend special classes with similarly intelligent children. The classes provided the ability to conduct independent projects. Gifted-program teachers encouraged their students to enter a series of science fairs then known as Olympics of the Mind, which challenged participants to solve unconventional problems. How do you build a tower out of just paper towel tubes? Propel a toy car with a mousetrap? Safeguard an egg dropped from an extreme height?

The experiences set Urmson up well to compete in Canada-wide science fairs. The year the Urmson family moved from Victoria to Trenton, the national finals happened to be held in Victoria. Urmson ached to visit his old friends, and so he directed all his energies toward winning the local competition. His entry, “Striking News About Impacts,” predicted the direction a body would travel after a collision. He won the Trenton fair, and received the free trip to Victoria.

Bit by the science bug, Urmson followed up with a project involving a model of ionic propulsion—“Ionic, Isn’t It?” was the project name. It not only won him another trip to the Canada-wide competition, but also garnered him second prize. Another year he won a silver medal at the national level and qualified for a four-week trip to study programming at Israel’s Weizmann Institute. Urmson would go on to study computer engineering at the University of Manitoba, where one of his projects entailed building a robot that traveled autonomously around a darkened room, seeking out the brightest sources of light.

Urmson was torn in his last year of university. One path, favored by mothers everywhere, might have seen Urmson going on to med school. Except that didn’t exercise his yen for building things, for envisioning complex systems and then figuring out how to make them work. Wandering by the office of his computer engineering department one day, Urmson’s eye was caught by a remarkable poster: a vehicle, maybe some sort of a planetary rover, climbing up and out of some sort of crater. “Come be a part of the robot revolution!” the poster read, with information about attending Carnegie Mellon University. It was a career based on the sort of thing Urmson had been doing all his life. Olympics of the Mind. Science-fair stuff. He applied, and ended up in Pittsburgh the following year.

At Carnegie Mellon, Urmson met Red Whittaker, who by 2003 already was a legend in American robotics and one of the best-known robot designers in the world. Born in 1948, Whittaker was fifty-five in 2003 and had become widely known for his willingness to take on projects that everyone else thought impossible. “If there is anyone in the world who can find a way to make things happen, it’s Red Whittaker,” said one colleague.

Whittaker may have been genetically programmed to ignore the impossible. His father was an air force bombardier in World War II who would go on to sell explosives to mining companies. His science-teacher mother was an amateur pilot who once flew under a bridge while the young Whittaker was in the plane alongside her. After serving in the marines for two years, Whittaker attended Princeton University, earning his degree in civil engineering in 1973, and then attending graduate school at Carnegie Mellon.

Whittaker made his name after the partial meltdown in 1979 of the Three Mile Island nuclear-generating station, America’s worst-ever nuclear accident. Cleaning up the incident required getting into the reactor’s basement to learn how radioactive the site was. Several contractors spent almost a billion dollars on the cleanup but still couldn’t figure out how to get inside. When Whittaker asked for his shot, the government figured they didn’t have anything to lose. Whittaker reasoned that, while the radioactivity prevented humans from getting into the reactor, machines should have no problem. He created a three-wheeled Remote Reconnaissance Vehicle, known as “Rover,” which he operated by remote control. Rover successfully made it to the basement. Best of all? The program cost only $1.5 million, which the government considered cheap.

Since then, Whittaker had specialized in building robots designed to work in harsh environments. One of his creations explored volcanic craters. Another mantis-like contraption built structures in space. Still another, created with a team that included the German software wizard Sebastian Thrun, crawled through the darkness of long-abandoned mines, mapping their interior passageways. Urmson had worked with Whittaker to develop computer algorithms designed to increase the speed at which robots were able to travel autonomously.

When Urmson returned from the Atacama Desert, he had a tough conversation with his wife, Jennifer. Urmson wanted to set aside the completion of his PhD for a time and pursue the DARPA race with Whittaker. The DARPA Grand Challenge was the talk of their academic specialty. DARPA had figured it might be lucky to get twenty entries. Eventually, 106 teams would enter. Urmson felt he had no choice but to join. Who knew what sort of fascinating epiphanies would emerge from the project? Who knew what Urmson would miss if he didn’t take part?

Urmson convinced Jennifer to let him do one race. The couple would put off having kids until Urmson was done. But then fate threw them a slider: It turned out Jennifer was already pregnant. The news added to the pressure Urmson felt to win. After all, it was the best way to ensure he’d get a high-paying job, once he finished.

To attract a team, Red Whittaker put up posters all around the Carnegie Mellon campus advertising an unconventional, graduate-level seminar class, Mobile Robot Development. It was pass or fail, and it featured only one assignment: to build a robot that would win the first DARPA Grand Challenge. He also sent out email blasts to potential sponsors and volunteers, which featured his trademark bravado: “The race defies prevailing technology, and many hold that the challenge prize is unwinnable in our time.”

Whittaker staged the first meeting of the team in a Carnegie Mellon seminar room on April 30, 2003, according to Wayt Gibbs, a reporter the magazine Scientific American embedded in Pittsburgh. “Welcome to the first meeting of the Red Team,” Whittaker began. “I am committed to leading this team to victory in Las Vegas next year.”

The men and women in the room were about as motley a crew as it was possible to put together in Pittsburgh tech circles. Bob Bittner was a former combat engineer who’d spent six years in a submarine. Spencer Spiker was a retired helicopter test pilot, a West Point–educated mechanical engineer who led two hundred people as a company commander in the U.S. Army, and who had left the service to spend more time with his family—then found himself jobless in a severe recession. He joined Red’s team because he had nothing better to do, then worked himself into a full-time staff position. Michael Clark was a NASA engineer who used a wheelchair to get around; having fallen on hard times, he lived for a spell out of his van. Lots of people had seen Red’s poster, apparently, and lots of people were inspired to work on the project it advertised. “I don’t know anything about computers—but I’d like to volunteer,” said Mickey Struthers, a postman who showed up to the first class because he wanted to participate in a historic science project.

“You’ve got a warm body”—Whittaker grinned, shaking Mickey’s hand—“and we need warm bodies.”

They began their efforts brainstorming what sort of a vehicle they would use. DARPA had announced the course would be designed by Sal Fish, the operator of such tough off-road races as the Baja 1000. Red Team figured it would have to be prepared for a course that would wind through dry river gulches, box canyons, mountain ridges, rocks, sagebrush and cliffs. So the robot they designed had to be able to either get around such land features or drive over them.

No idea was too outrageous to be considered. One of the first suggestions was a giant tricycle that had wheels seven feet in diameter. The team discussed using a Chenoweth combat dune buggy, a low-slung contraption on four fat tires favored by mercenaries and warlords. Other brainstorming options included construction equipment, an all-terrain vehicle and a tank. But the team ultimately opted for pragmatism. After all, Whittaker figured the budget to develop the robot would be around $3.5 million. Labor aside, $725,000 of that entailed the cost of the products required to build the vehicle. Whittaker was crisscrossing the country to find sponsors. Intel, Boeing and Caterpillar all kicked in some money. Google, which everyone thought of as a search engine company at this point, sponsored Red Team to the tune of $100,000 after Whittaker visited its headquarters in Mountain View, California, and met both Larry Page and Sergey Brin. But such funds wouldn’t go far when you were trying to build the fastest-ever robot car. Red had bought a cattle ranch a couple of hours east of Pittsburgh in the early nineties because he felt his academic life was too sedentary and sought physical activity that worked his muscles, not his mind. In September 2003, with the March 2004 race date fast approaching, Whittaker finally bought the vehicle that would become their robot from another farmer in the area.

Some in the class were astonished when they saw it. Shouldn’t a self-driving car look cool, and polished and, um, high-tech? The vehicle Red had procured was the opposite of high-tech. It was a High Mobility Multipurpose Wheeled Vehicle M998: a Humvee, battered by time. It was seventeen years old. No one had any idea how many miles it had, because the vehicle didn’t have an odometer. Nevertheless, the price was right: $18,000. The key thing was, it worked.

Whittaker was under a lot of pressure. Around the country, dozens of robotics enthusiasts were working to create entries for the challenge; so many, in fact, that DARPA was requiring everyone to submit a detailed and academically rigorous declaration of the approach they planned to take. The step was intended to limit the race entrants to serious competitors. There were high school students and bored mid-career engineers. Several were former contestants on the mechanical-gladiator game show, BattleBots, which featured remote-controlled robots fighting to the death, or at least, deactivation. Regardless of where they came from, the competitors all seemed to have one goal in mind: Beat Red Whittaker’s team. Why was the CMU team in so many other people’s sights? Whittaker’s team was the biggest, with thirty members. It was one of the best funded. And many also believed that it was DARPA’s hoped-for winner.

Red’s leadership style was to take a bunch of people, introduce the problem to them, set ambitious and clearly defined goals that reflected progress toward the solution—and then get out of the way. He’d drop in regularly to check in and apply pressure on his charges. Such visits could be intense. According to a Wired article, Whittaker once drew an analogy between developing robots and the labor required to construct the enormous historic monuments around the Nile. “If you’re in Egypt building the pyramids, you’ve got to have slaves,” he said. The implication? Whittaker’s students were his slaves. One of Red’s longtime students, Kevin Peterson, who would become Red Team’s software lead, had attended Princeton High School, where he encountered Dr. Anthony Biancosino, the domineering music teacher on whom Damien Chazelle loosely based the bandleader in the 2015 movie Whiplash. Peterson responded to Whittaker’s style, he says, in part because he’d already been through his experiences at Princeton with “Dr. B.” “There was an ethos around both of them of being larger than life and somewhat mysterious,” Peterson recalled. “The idea that you need to work hard to be part of their exclusive team if you want to join them. They’re both up to big things and you need to be a badass to be on the team. Funny thing is, both of them would accept and build anyone who had that level of dedication. It’s more about hard work than initial skill.” One of Whittaker’s favorite motivational anecdotes placed his charges in the roles of the Inuit in the Arctic, who had to decide which strategy to use when seeking out food. Are you going to go out and try to find a few berries and bits of lichen? Whittaker would ask his team. Or are you going to find and kill the walrus that feeds the whole village?

Sometimes it was hard to tell what Whittaker meant by his stories. Peterson interpreted this one as a challenge. Were you going to go about your life just getting by? Or were you the type who was going to go out and give your best effort to do something awesome?

Realizing that his course would require more work than they were prepared to give, some people dropped Whittaker’s class. The ones who remained essentially dropped every other one of their classes and just worked for him. Peterson was one of the ones who remained. He gave up his social life, as well as communicating with his family. He even gave up sleeping. Several months in, he became so sleep-deprived that he fainted. The problem was that he was going down a set of stairs when he did. He hit his head, was taken to the hospital to be assessed—and was back working on the project within a few days.

Empowering inexperienced and sleep-deprived graduate students who were totally committed to the project’s success could create some unusual situations. One morning, Whittaker and Urmson arrived to check in on the students and volunteers and were met with the results of one of these hyper-caffeinated work sessions: Their treasured Humvee no longer had a roof. Working through the night, one of the student team members had decided that the Humvee’s interior didn’t have enough room to store the batteries and computers and actuators that the self-driving equipment would require. So he went and got a Sawzall and cut through each one of the Humvee’s roof pillars, essentially decapitating the vehicle.

This was the sort of initiative that would typically have been applauded by Whittaker. Except the impromptu roof amputation wasn’t really necessary. Even if the equipment couldn’t fit in the Humvee’s cab, they could have ripped out some seats, or mounted additional equipment on the Humvee’s roof. Removing it made the vehicle illegal to drive on public roads. From then on, whenever they wanted to take the Humvee to the sort of wide-open space where they could test it, they would have to tow the vehicle—an ignoble start for a robot that was supposed to drive itself.

To provide the Humvee with the ability to drive itself, Red Team essentially reverse-engineered the sensory tools humans use to help them drive. The vehicle needed, for example, eyes to see—and so the Red Team procured several types of LIDAR (Light Detection and Ranging) devices. The LIDAR’s job was to shoot out beams of light and sense when the beams bounced back. Precisely calculating the timing of the beams’ return allowed the LIDAR to determine how close the sensor was to the object that the light beam bounced against. Repeated thousands of times per second, the LIDAR could create a rudimentary picture of the world outside the vehicle.

The main LIDAR sensor would allow the robot to detect obstacles seventy-five meters ahead. Three supplemental LIDAR devices scanned a wider field of view within twenty-five meters of the robot’s front end. A stereo-vision processing system represented a different way to use light to detect objects, employing a pair of cameras. But the cameras and LIDAR might have trouble penetrating the dust clouds that could arise on sandy desert roads. To provide a sense of the world in dusty conditions, Red Team also bought a radar system that used sound to detect obstacles.

To control the vehicle’s direction and speed, Red Team wouldn’t be able to use a foot on the gas pedal or a hand on the steering wheel. Actuators would take their place. Essentially, these were electric motors that twisted, pushed or pulled—to make the vehicle accelerate, brake or turn left or right.

Sitting in the center of all that was a series of computers, the robot’s brain. Donated by Intel, one was a quad-processor Itanium 2 server that featured three gigabytes of RAM. Some of the computers were intended to combine the information provided by the LIDAR, the stereo-vision system and the radar sensor to create a model of the world. Another computer employed GPS data and motion-tracking tools to locate the robot in the world within a single meter of accuracy. Now that it had a conception of its surroundings and knew its location, the robot’s computer system would have just two questions to answer. Two questions that humans asked themselves, thousands of times a trip: How fast should I be going? And where should I be steering?

Whittaker scheduled one hundred days to actually get the robot assembled and the software built. The deadline fell in November, but as Thanksgiving approached, significant portions of the vehicle remained unfinished. The computers weren’t wired together, for example. Nor were the sensors mounted. The robot did have a name, though: Sandstorm, after the dust clouds the vehicle would kick up in the Mojave.

Whittaker and Urmson both worried a lot about the Mojave Desert. They worried about the off-road conditions of the course, and the effects of the Mojave’s rutted roads on their sensitive sensors and microprocessors. Driven over even at moderate speed, the Mojave Desert’s rocks and ridges were bound to create vibrations that the students believed had the potential to damage the computer’s memory. After all, your basic disk drive is just a magnetic metal plate that spins really quickly. They’re encoded by a precise bit of metal that hovers just above the plate. Extreme shocks could see the metal stick gouging chunks from the spinning plate and damaging the drive. Those same bumps could create false readings from the sensors.

Consequently, Red Team spent a lot of time determining how to insulate the computers and sensors from the jars and bumps that would happen as the Humvee drove across the desert. The solution, they decided, was to protect the equipment the same way automobile manufacturers insulated humans from bumps and jars. With springs and shock absorbers, which were fitted to an enormous metal box where the Humvee’s roof used to be. Dubbed the “e-box,” for electronics box, the 1,200-pound container didn’t just contain hard drives. It also encompassed much of the robot’s most sensitive equipment—the computers, the GPS system, the radar as well as the supplementary LIDAR units.

The main LIDAR and the stereo-vision device still remained sensitive to the pitches and rolls that could strike the robot as it navigated the off-road trail. So the team spent untold hours engineering a device based on old nautical gimbals, complex series of interconnected arms and pivots that kept a ship’s compass stable in even the heaviest of seas. Part of the Red Team designed and built their own gimbal, mounting inside it the main LIDAR and the stereo-vision system, and protecting it all in a sphere a little larger than a classroom globe. Little motors in the gimbal allowed Sandstorm to direct the LIDAR and camera wherever it needed to sense the world. Heading into what its onboard map told it was a leftward curve, the LIDAR would “look” to the left so that it could see the world in the direction of the world to come.

As technical director, Urmson was the one in charge of putting all these pieces together. He felt enormous pressure both at home and on the Red Team. That September, his wife had just had the couple’s first child, a baby boy. But Urmson couldn’t spend much time at home. He had made a promise to Whittaker that the robot would drive itself the entire length of the race, 150 miles, by midnight on December 10, 2003—three months before race date.