Autonomy: The Quest to Build the Driverless Car - And How It Will Reshape Our World

To meet that deadline he was working sixteen-hour days, seven days a week; during one furious round of assembly Urmson didn’t sleep for forty hours. The week before Thanksgiving, Whittaker added to the pressure. “This vehicle hasn’t rolled so much as a foot under its own control,” he said during one meeting with Urmson and other key team members, according to the journalist Wayt Gibbs. “You have promised to get 150 miles on that beast in two weeks … Anyone who thinks it is not appropriate for us to go for 150 miles by December 10, raise your hand.” Silence. Not a single person elevated an arm. Whittaker smiled, according to Gibbs, and made an observation in his characteristically florid language: “We’re now heading into that violent and wretched time of birthing this machine and launching it on its maiden voyage.”

The assembly work happened in a big garage in Carnegie Mellon’s Planetary Robotics building. Envision the best mechanics shop you’ve ever seen, and you’ll be close to this workspace. The ceiling is a few stories tall, with gangways and a small-scale version of a crane, the better to lift heavy objects. Lathes and drill presses, drawers full of every implement imaginable, as well as computer diagnostic equipment—every available horizontal surface features tools. It is the kind of place where you could literally make almost anything.

The venue would host Urmson and the members of his team pretty much nonstop through that Thanksgiving weekend. By the end of it, enough computers were wired together, and enough sensors mounted, that Sandstorm felt like it was coming alive. It was around this period that the team found the perfect place to test their Frankenstein’s monster. There weren’t many spots with convenient access to the CMU campus where a 5,000-pound, exhaust-snorting, diesel-gulping, oil-dripping robot could push the limits of its abilities without risking civilian fatalities. It was Mickey Struthers, the postman volunteer, who thought of the solution. One day while he was driving over Pittsburgh’s Hot Metal Bridge on the way to Carnegie Mellon, Mickey noticed the lights along the shores of the Monongahela River twinkling in the cool evening air. All except for a vast swathe of dark shoreline to the right of the bridge. Mickey knew that was industrial land that had once housed Pittsburgh’s last steel mill, the LTV Coke Works, which had closed in 1998. Since then the land had sat fallow.

Struthers suggested the site to Whittaker, who loved the idea for both its convenience as well as its industrial heritage. The 168-acre land parcel housed a railroad roundhouse and numerous outbuildings and equipment that made it seem as though it was left over from the industrial revolution, connecting the team to the same brawny spirit that had built Pittsburgh so many decades ago. With a few phone calls to the wealthy family foundations that owned the land, Whittaker arranged for the team to test there.

On the second of December, the team took the first of what would become many test runs at the Coke Works. The distressed location with its spent oil cans and rusted industrial detritus seemed appropriate for the ancient-looking Humvee, which just in general seemed to have more in common with a Jurassic-era dinosaur than one of the most innovative mechanical devices ever assembled. Snow covered the ground. The temperature was eighteen degrees. “Just like the Mojave Desert, huh?” shouted one team member, according to a Wired article. (Whittaker, meanwhile, was wandering around in a knit shirt, jeans and boots he wore without socks.) Urmson climbed aboard for the first run to manually hit the emergency stop button if the robot suddenly went crazy. The robot swerved toward a precipice when first activated, then settled and drove its course as expected. After a few uneventful laps Urmson decided, at 7:51 P.M., to see what would happen when he gave Sandstorm free rein. He clambered off the robot. The team programmed in a series of GPS waypoints that drew a dot-to-dot version of an oval. Not sure whether to breathe, the team watched the robot roll along its route for half an hour, ultimately accumulating four miles. No accidents. No incidents of any kind, in fact. They were nowhere near making their 150 miles yet, but that evening, it was difficult to deny they were progressing toward their goal.

Another week passed, and late in the evening on the tenth of December, with just a couple of hours before the midnight deadline by which Urmson and the team had promised Whittaker that Sandstorm would be able to drive 150 miles on its own, the robot was not cooperating. Bugs arose in the self-driving software every time it drove more than a few laps. Urmson and his fellow teammates had been camped out for days at the Coke Works, if you called camping sleeping in your running car with the heat on full blast. Despite daylong debugging sessions, Sandstorm remained unpredictable and occasionally suicidal—lurching into a telephone pole, catching fire, becoming suddenly unable to sense GPS signals. A calm spell saw the Humvee revolving the track, again, again, again, and then for no apparent reason, swerving off course and running itself through a chain-link fence before Urmson could activate the e-stop. Sometime later, with Sandstorm liberated from the barbed wire and the deadline approaching, Whittaker gathered Urmson and everyone else around him, according to Gibbs. Sure, the December 10 deadline approached—but even if it passed, Whittaker vowed, they’d continue their work, through tomorrow, and even the next day if necessary, until Sandstorm achieved the 150-mile goal. “We say what we’ll do, and we do what we say,” vowed Red in Scientific American.

Then it started to rain—a frigid December drizzle that soaked clothing and chilled to the bone. Sandstorm was not well protected against rain. One of the dozen or so team members still on site spread a tarp over the robot’s computer equipment. Red wasn’t around. Gibbs wrote that Urmson looked at his teammates, shivering in dripping lean-tos under blankets. He thought about the possibility of the falling moisture disabling one of their sensors, or shorting out a processor. Perhaps he also thought about his wife and baby boy back home. And he decided to send the team home.

Whittaker was livid when everyone showed up to the Coke Works the following day, Gibbs reported, comparing the team leader to “an angry coach at halftime.” He ranted about all the sacrifices they’d made to try to achieve the 150-mile goal. The shop was a mess, the robot unpainted, the website out of date—all that work went undone as everyone concentrated on getting Sandstorm in the sort of shape required to make its race run. To a roomful of people unwilling to meet his gaze, Whittaker said, “Yesterday we lost that sense deep inside of what we’re all about. What we have just been through was a dress rehearsal of race day. This is exactly what the 13th of March will be like. We’re in basic training; this is all about cranking it up a notch. Come March, we will be the machine.” Whittaker concluded his venting, Gibbs reported, by asking who was willing to work all day, every day, for the next four days, until they completed their nonstop 150-mile run. Fourteen team members in the room raised their hands. Including Urmson.

Two days later, U.S. soldiers captured Saddam Hussein in a spider hole near Tikrit, and the war in Iraq dominated headlines and the cable news channels as it never had. Every day, the news seemed to feature more casualties from IEDs in Iraq or Afghanistan—fatalities Red Team members hoped the robot vehicles might one day prevent. Then the overseas conflicts supplied Urmson with an idea.

In recent years, maps had become a crucial component of successful robotics. Maps allowed robots to locate themselves in the world much more accurately than GPS alone. A technique called simultaneous localization and mapping, abbreviated to SLAM, saw a robot scan an area with LIDAR to map the permanent landmarks—in exterior spaces, things like trees, light poles, road curbs and buildings. Then, the next time the robot traveled the same territory, it would consult its map and compare its position relative to the previous landmarks, to get an ultra-accurate idea of where it was. Problem was, Sandstorm couldn’t use this technique, because DARPA was keeping the race location secret.

Then, one day, Urmson was watching coverage of the war on one of the cable news channels. The scene will be familiar to anyone who lived through the post-9/11 period—a grainy portrait of an SUV traveling fast along a remote desert road. From somewhere in the distance, a rocket blazes into the picture, collides with the SUV and obliterates the vehicle in a blast of dust and metal.

The footage of the successful deployment of a laser-guided bomb was captured by a camera-equipped drone aircraft. The drones flew above the conflicts to provide imagery of the Iraqi and Afghan territories. Drones were searching Afghanistan for Al-Qaeda hideouts that might shelter Osama bin Laden. They were scanning Iraq for nests of Ba’athist loyalists.

If the U.S. military could use drones to obtain imagery of places so hostile and remote, Urmson thought, then such imagery would soon be available for the entire world. And perhaps, Urmson reasoned, that same type of imagery could be used to simplify the robot’s task. They weren’t able to use LIDAR to scan the race course in advance, because no one on Red Team knew where the race course was, but they did know the race went across the Mojave Desert—and maps existed of that, didn’t they? In fact, portraits of the Mojave had already been built by entities like the U.S. Geological Survey and the military.

“We realized we didn’t have to do SLAM,” Urmson recalled. “Because it was becoming clear there would be a global database [of maps] available … So why not use them?”

If Red Team members could give Sandstorm an accurate map of its surroundings before the race, they could remove a time-intensive step from the computational task. The new approach reframed the challenge. The team had assumed they were trying to build a robot that could sense the world so well, it could discern a road in the desert and navigate it safely for 150 miles. Using maps meant the robot could be told in advance where the road was, and how to drive it. The method had the potential to allow Sandstorm to travel much faster than it otherwise might.

But first, Red Team’s undergraduates, pauper grad students and volunteers would have to build the most detailed map of the Mojave Desert ever assembled. It was an enormous task, but Red Whittaker’s students were accustomed to achieving enormous tasks. A portion of the team set to procuring high-res maps of the whole of the Mojave Desert, a relatively simple matter, given Whittaker’s and Spencer Spiker’s defense contacts. Now the team set about using the maps to plot routes through the Mojave. They also dispatched two engineers, Tugrul Galatali and Josh Anhalt, to drive as many roads in the Mojave Desert as possible in a rented SUV with video cameras sticking out the windows, capturing imagery from the ground in what amounted to an early, rudimentary execution of Google’s Street View idea.

The next step saw the Carnegie Mellon mapping team comparing the footage and the map to assign each area with a value—what they called a cost. So a ridge or a cliff that would wreck Sandstorm if the robot went over it would get a cost of infinity. A smooth road or a dry, flat lake bed likely would have a cost of zero. Sandstorm’s computers then were programmed to direct the robot to drive the route with the lowest cost.

One evening, with just weeks to go before race date, the senior members of Red Team met in the loft of Carnegie Mellon’s Planetary Robotics building. “We were making some progress, trying to map every trail in that whole desert,” Urmson recalls. But at some point during this meeting in the loft, Urmson realized their work wasn’t happening quickly enough. “It became clear we weren’t going to get there,” he said. Too many different potential routes existed. By the time the race date arrived, they would have mapped out only a small portion of the possible routes.

That was the point that Red Team came to its second epiphany. To reduce the possibility of exactly this sort of advance route planning, DARPA had told the teams that its staff would wait to disclose the precise course until just two hours before the start—at 4:30 A.M. the morning of the race. Red Team was getting good at creating routes through the desert. So what if they changed strategies? What if, rather than focusing on creating a map that featured a pre-driven route along every single conceivable trail through the desert, they instead became really good, and blindingly fast, at teaching Sandstorm to drive a single trail?

Rather than a perfect map, they thought, why didn’t they focus on creating a single, perfect route? One they could plan out in the two-hour span between the time DARPA disclosed the approximate course and the start of the race? The old way involved using the maps and the route planners during the months before the race to effectively pre-drive every single road through a desert that covered a territory of fifty thousand square miles. This new way involved focusing on a single 150-mile path that the planning team would examine in fine detail—and doing it in the 120 minutes that passed after DARPA disclosed the race route.

From that moment on, one part of Red Team focused on executing the second epiphany. In the old high bay in the Planetary Robotics building, about a dozen members rehearsed exactly what would happen after DARPA handed over the route in a computer file at 4:30 A.M. The file would feature a series of about 2,500 GPS waypoints, which everyone referred to as “breadcrumbs,” spaced about a hundred yards away from one another, tracing out the course in a dot-to-dot fashion. The dozen members of Red Team’s planning unit would leap into action. One would feed the file into a software program that used the Mojave map’s cost estimates to build a more precise route, with many times more breadcrumbs than DARPA’s route network definition file (RNDF).

But Urmson, Whittaker and their team didn’t trust the route calculated by the planning software. It had been known to send Sandstorm on journeys that went over ridges, into ditches or through wire fences. So a team of editors would divide up the course into sections and then, using computers, virtually go over every yard of the computer-calculated race path to make sure the software hadn’t made any mistakes. Once the human editors were done correcting the course, they’d reassemble it into a single route and upload it to Sandstorm, to execute on the race course.

Still, by January 2004, just two months before race date, Sandstorm had not yet gone fifty miles on its own. One thing causing Whittaker and Urmson anxiety was the disconnect between where they were testing Sandstorm and the race course. They were testing the robot on the frigid shores of Pittsburgh’s Monongahela River. The race would be held in the Mojave Desert. Would the change in environment pose a problem to Sandstorm?

In February, Whittaker arranged for some of the team’s key members, including Urmson, Peterson and Spiker, to accompany Sandstorm to the Mojave Desert to refine the robot’s capabilities. (Sandstorm actually made the trip in a fifty-two-foot enclosed semi-trailer.) The final part of preparations would happen at the Nevada Automotive Test Center, an enormous swathe of desert where companies from all parts of the automotive sector, from tire manufacturers to transmission firms, tested their products in the harshest desert terrain available.

In Nevada, Urmson’s team worked exclusively on Sandstorm. Write code, take Sandstorm out to test the code, watch for mistakes, take note of the mistakes, write code. They repeated the cycle without regard to clocks or arbitrary separations of day and night. Two, three days at a time they worked without sleeping, fueled by Mountain Dew, Red Bull and junk food, and then, when they were too exhausted to manage to keep themselves vertical, they slept. Sometimes in an RV they’d rented, although the trailer didn’t have enough beds for all of them; others slept on the floor of the test center’s mechanics shop on folding lawn chairs, or in the reclined seats of the SUVs they rented to tail Sandstorm.

Working nonstop, through night, through day, the way they did presented some difficulties. One evening, past midnight, Sandstorm ran into a fence post, wrecking the front bumper, which was necessary to support cameras and radar sensors. The test center’s mechanics building was locked up, of course, but in the spirit of asking for forgiveness being easier than requesting advance permission, Spiker and one of the students scaled the fence and broke into the building, where they welded together an entirely new bumper with thick steel pipe. The thing ended up weighing about two hundred pounds—making it more than able to support the sensing equipment the robot required. “You could probably have driven through a building and not hurt that thing,” Spiker recalls.

One thing they didn’t do much of was bathe. The wastewater tank in their rented RV filled up, and by the time they got around to driving it to the nearest town to empty it, the vibrations from the washboard dirt road into town splashed sewage all over the RV’s interior. Cleaning the mess was so traumatizing that the team outlawed use of the RV’s bathroom. While there were bathrooms available in the mechanics shop, no other showers were available, so the guys went without washing for about six weeks. Then, in mid-February, one of their computer sponsors, Intel, invited the Nevada members of Red Team to San Francisco, where the computer chip manufacturer wanted to show off Sandstorm at the Intel Developer Forum.

By that time, Sandstorm had managed a speed of 49 mph and an autonomous run of a hundred miles. The guys were excited about the progress they’d made. But the robot still had its mechanical idiosyncrasies. It was apt to see obstacles that weren’t there, or miss obstacles that were, or even misinterpret pre-programmed commands. What if something like that happened while Sandstorm was onstage at the conference?

The following morning, an audience of hundreds watched the autonomous vehicle creep out onto the stage, apparently thanks to the benefit of high-tech sensors, engineering and computers powered by “Intel inside.” The crowd cheered in response. The applause felt good to the Red Team members present. Here they were at a Silicon Valley event being treated like celebrities. The recognition validated their sacrifices and the worth of the project. It also made the team thankful that no one realized that during the onstage demonstration, a Red Team member had hidden in the space under Sandstorm’s steering wheel, prepared on a moment’s notice to slam his hand on the brake pedal if the massive robot threatened to roll off the stage into the crowd.

On Friday, March 5, 2004—eight days before the race and just three days to go before the qualifying events—Chris Urmson rose early in the morning, put on his usual uniform of a mud-spattered baseball cap, fleece sweater and worn jeans, laced up his running shoes and decided that today would be the day to stage Sandstorm’s culminating test challenge.

Urmson, Peterson, Spiker and the rest of the Nevada squad tested Sandstorm in the worst conditions they could imagine—frequently, along sections of the trail the old Pony Express had followed more than a hundred years earlier. “Red is really gung-ho about testing hard,” explains Peterson. DARPA had said its route would be about 150 miles. The longest run Sandstorm had made was a hundred miles. But with the race a little more than a week away, everyone on the team was hoping for a longer run to boost their confidence.

The goal was just like the race: 150 miles in ten hours. The route amounted to a flat oval, about two miles around. While they prepared Sandstorm, Urmson and Peterson tinkered with a new part of the software: a component of the speed-setting module designed to slow down the robot when it approached a curve. The new code was designed to allow Sandstorm to drive more quickly on straightaways.

The code worked wonderfully. During a few warm-up laps, Sandstorm managed to get up to 49 mph along the straightaways and then the new algorithm slowed it down as the robot headed into the curves. In fact, as Urmson and Peterson watched the robot, they wondered whether it slowed Sandstorm too much. An adjustment to the algorithm during a refueling break seemed to improve things. On the first lap they watched as Sandstorm cruised into a curve, slowed a little bit and then accelerated through the curve’s exit. At the end of the second lap, Sandstorm was heading fast into what Urmson would later describe in his field test report as a “soft S-curve” to the left. The right-side tires drifted off the road into deep sand, and when Sandstorm tried to correct things, to get back on the track’s packed-down dirt, it steered too hard to the left. The right-side tires bit into the soft sand. The left-side tires came up off the road. Behind, in the chase car, Urmson watched, horrified, as Sandstorm tipped up and over, and came to rest upside down—right on top of the e-box and the gimbal housing all the vehicle’s most sensitive equipment.

The robot had been designed to insulate the box’s components from being damaged in all sorts of accidents. Front-end collisions, rear-end collisions—pretty much any collision that happened on the ground plane, Sandstorm would be able to withstand just fine. But the robot had one fatal weakness: a rollover. Because Humvees sat comparatively low and flat, their geometry made rollover accidents almost impossible.

Unless you were testing a robot Humvee in the Mojave Desert, apparently.

A History Channel crew had come out to film the test run. They rushed out onto the track with their cameras and shoved one into Urmson’s face, asking him to list the damage. Urmson looked at the wrecked robot the team had spent the better part of a year engineering: at the crushed gimbal, the compacted GPS antennae, the flattened e-box and the connecting rods bent out of shape. And he let fly with the expletives that made him one of the few people to ever have to have been bleeped by the History Channel. “Shock and disbelief,” Urmson says when asked to describe his reaction, more than ten years later. “But mostly disbelief.”

Disbelief, because they had felt like they’d been making such great progress. Disbelief because they were just days from the qualifiers. Disbelief because this had happened on the second lap of a two-mile track that Sandstorm was supposed to drive for seventy-three more laps.

Most of the crew figured Red Team was over. That they’d never be able to repair the robot in time. Somebody called Pittsburgh to inform everyone else about the accident. Red’s assistant Michele Gittleman took the call. She recalls sobbing when she processed the news.

Maybe a crew led by someone other than Red Whittaker would have given up. But Whittaker didn’t even consider it.

At the Nevada Automotive Test Center, Urmson, Peterson, Spiker and the rest of the team attached the four-by-four chase vehicle to Sandstorm with the help of some nylon webbing and managed to flip over the robot. Spiker, the most mechanically minded among them, went over the engine to look for problems. The engine was flooded with diesel fuel but aside from that, everything looked okay.

The other guys assessed the electronics equipment. The GPS units were toast. The gimbal suffered the worst impact and would need to be completely rebuilt. The main LIDAR unit was irreparable. Luckily, an extra gimbal and LIDAR sat in storage back in Pittsburgh. They towed Sandstorm to the mechanics shop and for three nearly sleepless days and nights they worked to fix everything they could. And they nearly did it.

The race was March 13, 2004. Heading into the qualifiers the week before, at the California speedway in Fontana, the GPS system worked, which meant the robot could locate itself in the world. The sensors were active, which enabled Sandstorm to perceive obstacles. The computers could calculate the trajectories required to follow the path set out by the Red Team mapping crew. The only problem? “We had no time to calibrate,” recalls Whittaker. Which meant Sandstorm viewed the world through a distorted lens.

Think of each sensor as its own individual eyeball. You are able to see one version of reality because your brain is able to amalgamate the view from your two eyes into a single picture of the world. Sandstorm amalgamated the information from four different LIDAR units plus the stereo-camera system. Ensuring that the robot’s sense of the world resembled the actual world required calibrating the individual sensors—a time-intensive process of trial and error. “Think of calibration as alignment,” Whittaker explains. “Even in the car shops they align your headlights, right? And when there are multiple sensors that will fuse data into a common model, it’s important that they’re all aligned. If you just bolt it back together you’re creating a kind of Frankenstein, and maybe it’s a little cross-eyed.”

And so the cross-eyed Frankenstein’s monster limped into race week, belching diesel exhaust, dented and scratched, but otherwise intact. Red Team would compete against twenty other entries from across the United States. In the qualifiers each of the twenty-one robots would have to navigate a mile-long obstacle course to progress to the main event.

Soon after his arrival at the California Speedway, Urmson wandered around to see what he could learn about his competition. He saw Doom Buggy, created by the only high school admitted to the competition, Palos Verdes, near Los Angeles. An undergraduate from UC Berkeley, Anthony Levandowski, led the team behind the only two-wheeled entry, a robot motorcycle that was able to balance itself with the help of a gyroscope. UCLA’s entry, the Golem Group, was led by a guy named Richard Mason, who had seeded his project with $28,000 he’d won on Jeopardy!

On the other end of the spectrum were the professional teams, which were affiliated with various engineering-focused corporations. An inventor named Dave Hall had created an autonomous Toyota Tundra pickup truck that was notable for driving smoothly with a stereoscopic camera setup—it didn’t use any LIDAR at all. From Wisconsin, the makers of Oshkosh Trucks entered a six-wheel-drive, 32,000-pound fluorescent yellow behemoth with the imposing name of TerraMax. Louisiana’s Team CajunBot also used a six-wheeled vehicle. A fraction the size of the Oshkosh entry, it was based on an all-terrain vehicle more commonly used by the state’s hunters to navigate the bayou.

Urmson, Peterson, Whittaker—all of them wandered the event, talking to people just as technically minded as they were. It quickly became apparent that Red Team was among the biggest of the teams. Popular Mechanics gave them seven-to-one odds to win, highest of all their competitors’. The front-runner status positioned Red Team as the entry everyone else wanted to beat. Urmson had posted on the Red Team website a photo of Sandstorm after the rollover. Now, as he wandered the raceway, talking to the leaders of other teams, Urmson spied the photo on numerous computer monitors. Some other teams had made it their wallpaper—as motivation.

Adding to the excitement was the fact that DARPA’s public relations team had arranged for reporters and television producers from across the nation to visit the raceway. Urmson and his teammates had toiled for months in obscurity. The accolades they received at the Intel event had been nice, but the more common reaction to their work was incredulity. “A car that drives itself?” people would scoff. To many, it sounded ridiculous. The presence of the reporters going around interviewing anyone available reminded the competitors that their work was important. Important enough for the U.S. government to put up a million-dollar prize. Important enough, possibly, that it might save the lives of U.S. soldiers fighting in distant theaters of war.

On the morning of March 13, 2004, the start of the race was one of the most exciting moments Chris Urmson had ever experienced. The robots were lined up in their starting chutes. Media and military helicopters hovered in the sky. Grandstands supported hundreds of spectators, each of them getting whipped by the desert sand, and over it all, Tony Tether’s amplified voice marked the momentous event.

“We’re thirty seconds from history,” shouted the DARPA director into the microphone. “All right, ladies and gentlemen, boys and girls, the bot has been ordered to run, the green flag waves, the strobe-light is on, the command from the tower is to move!”