Smart Swarm: Using Animal Behaviour to Organise Our World

“If I was in charge of designing the software for a company like Air Liquide, I’d probably be stressed about doing a really great job,” Gordon says. “But the ants aren’t doing that.” Their system’s too loose and undisciplined. Information coming in is too spotty, and their responses are too unpredictable. “The amazing thing to me is how, every way you look at it, the ants’ system is so messy, and yet somehow it works,” she says.

Maybe there’s a deeper lesson here, Gordon suggested. “Instead of trying to keep fine-tuning a system so it will work better and better, maybe what we really ought to be looking for is a rigorous way of saying, okay, that’s good enough.” Maybe a smart way to face the unpredictable, whether you’re running a business or playing a game of checkers, is to look for that balance between strategic goals and random experimentation. Ant colonies, after all, manage to thrive at the edge between efficiency and utter chaos, she says. “The question is, how do they find that edge? Because if we could find that edge too, we could save ourselves a lot of trouble.”

2 HONEYBEES Making Smart Decisions (#ulink_18b2ff19-1cbe-5e6c-a88f-e8391a20a7d4)

Appledore Island is a tough place for honeybees.

Anchored in the Atlantic off the coast of southern Maine, the rocky, wind-blown island is barely a half-mile long, with hardly any trees, which the bees need for nest building. In fact, you might describe the island as a kind of bee Alcatraz, which makes it an ideal place to observe their behavior under controlled conditions.

A few summers ago, biologists Thomas Seeley of Cornell University and Kirk Visscher of the University of California at Riverside ferried a half-dozen colonies of honeybees to Appledore, which is home to the Shoals Marine Laboratory run by Cornell. For nearly a decade, Seeley and Visscher have been studying a fascinating example of what they call “animal democracy.” How do several thousand honeybees, they want to know, put aside their differences to reach a decision as a group?

The focus of their research has been honeybee “house hunting.” In late spring or early summer, as a large hive outgrows its nest, the group normally divides. The queen and roughly half of the bees fly off in a swarm to create a new colony, leaving behind a daughter queen in the old nest. There may be fifteen thousand bees in the swarm, which typically clusters on a tree branch, while several hundred scout bees search the neighborhood for new real estate. Although the queen’s presence is important to bees in the swarm, she plays no role in picking a new nest site. That task is delegated to the scouts, who do their jobs without direction from a leader.

When a scout buzzes off into the countryside, she’s looking for just the right dwelling place (I say she, because worker honeybees are all females). It must be well off the ground, with a small entrance hole facing south and enough room inside to allow the colony to grow. If she finds such a spot—a hollow in a tree would be perfect—she returns to the swarm and reports her discovery by doing a waggle dance. This dance, which resembles the one forager bees do when they locate a new patch of flowers, contains a code telling others how to find the site. Some of the scouts that see her dance will then go examine the site for themselves, and, if they agree with her assessment, they’ll return to the swarm and dance in support of the site, too.

This is no trivial question for the bees. As long as the swarm is clinging to the branch, it remains exposed to weather, predators, and other hazards. But once the swarm selects a new home, it won’t move again until next spring. So it has to get it right the first time. If the group selects poorly, the entire colony could perish.

One by one, scouts that have been exploring the neighborhood return to the cluster with news about different locations. Soon there’s a steady stream of bees flying between the cluster and a dozen or more potential nest sites, as more and more scouts get involved in the selection process. Eventually, after enough scouts have inspected enough sites, it becomes clear that traffic at one site is much greater than that at any other, and a decision is reached. The bees in the main cluster warm up their wings and fly off together to the chosen site—which almost always turns out to be the best one.

Facing a life-or-death situation, in other words, a honeybee swarm engages in a complex decision-making process involving multiple, simultaneous interactions between hundreds of individuals with no leadership at all—exactly the kind of chaotic, unpredictable enterprise that, if attempted by people under stress, would almost certainly lead to disaster. Yet the bees almost always make the right choice.

How do they do it?

The Five-Box Test

One spring day in 1949, a young zoologist named Martin Lindauer was observing a swarm of bees near the Zoological Institute in Munich, Germany, when he noticed something odd. Some of the bees, he realized, were doing waggle dances. Ordinarily that meant they were foragers that had found a nice patch of flowers nearby, and they were telling other bees where to go find it. But these dancers weren’t carrying any pollen or nectar, so Lindauer didn’t think they were foragers. What were they up to?

Lindauer’s mentor at the University of Munich, the renowned zoologist Karl von Frisch, had recently figured out that the waggle dance—or “tail-wagging dance” as he called it—was in fact a sophisticated form of communication (he won a Nobel Prize for this research in 1973). When a foraging bee danced, von Frisch had discovered, she wasn’t just advertising a source of food, she was also providing precise directions to locate it. To perform such a dance, a bee would run forward a short distance on the hive’s comb while vibrating her abdomen in a “waggle.” Then she’d return to her starting point in a figure-eight and repeat this over and over, as if reenacting her flight to the flower patch. The length of her dance indicated how far away the food was, and the angle of her dance (relative to vertical) corresponded with the direction of the food (relative to the sun). If a bee danced in a direction thirty degrees to the right of vertical, for example—picture the number 1 on a clock—the flower patch could be found by flying in a direction thirty degrees to the right of the sun. It was an ingenious system, but it had never been linked to house hunting before.

By carefully studying several swarms—sometimes running beneath them as they flew across the Bavarian countryside—Lindauer determined that the bees dancing on the swarm cluster were scouts that had been out searching for a nest site. Some were still powdered with red-brick dust from having explored a hole in a building, or blackened with soot from having checked out a chimney. Just as foraging bees used the waggle dance to share news about food sources, so the scouts were using it to report on potential real estate. At first, many of the scouts danced in different directions, apparently announcing various options. But after some hours, fewer and fewer sites were mentioned until, finally, all the dancers were pointing in the same direction. Soon after that, the swarm lifted off from its bivouac and flew to its new home, which Lindauer was able to locate by reading the code of the dances.

The bees had reached a consensus, he theorized, because the liveliest scouts had persuaded the rest to go along with their choice. They did this by getting rivals to visit their preferred site, where, confronted with the superior qualities of the site, the former competitors simply changed their minds. One by one they were won over, he speculated, and the disagreement went away.

In this respect, at least, Lindauer got it wrong. It wasn’t quite that simple. Researchers have since established that only a small percentage of scouts ever visit more than one site. The group’s decision does not rely on individual scouts changing their minds, but rather on a process that combines the judgments of hundreds of scouts—one that would remain a mystery for fifty years.

That’s where Tom Seeley and Kirk Visscher came in. Beginning in the late 1990s, they picked up where Lindauer left off, this time using video cameras to record every aspect of the swarm’s behavior. They also brought some new ideas about honeybee deliberation. Given the large number of individuals that take part in house hunting, they doubted that bees’ decision making was based on consensus. It just seemed too complicated, like trying to get a large group of friends to agree on which movie to watch. More likely, they figured, the process relied on some form of competition. Instead of trying to work through their differences with one another, scouts dancing on the swarm cluster appeared to be actively lobbying for different sites. It wasn’t a meeting of minds at all, but a race to build up supporters—with the winners taking all.

In that sense, the bees’ system was more like a stock market, in which the value of a security rises or falls according to the collective judgment of the group. Scouts watching another scout dance, like brokers, might be persuaded to do their own research on the site being advertised, and if they liked what they saw, they could buy into the site by dancing for it themselves. If they didn’t like it, they didn’t have to. The more bees that joined in, the greater the likelihood the site would be selected.

But how did the process work, exactly? What were the mechanisms that enabled the bees to choose so accurately?

To find out, Seeley and Visscher conducted a series of experiments. After preparing a swarm for house hunting, they placed five plywood nest boxes an equal distance from the bees on Appledore Island—four representing mediocre choices for a new nest and one that was excellent. What made the fifth box better than the rest was that it offered the bees an ideal amount of living space—about forty quarts, compared to fifteen quarts for the others, which was not enough to store honey, raise brood, and meet the other needs of an expanding colony. To track the bees during the decision-making process, Seeley and Visscher labeled all four thousand individuals in each swarm with tiny numbered disks on their thoraxes and dabs of paint on their abdomens, a tedious process that involved chilling batches of twenty bees at a time to render them docile enough to be handled. But it was worth it in the end, because, when they looked at video tapes of the swarms later, they could tell which bees had visited which nest boxes and which ones had danced for which boxes at the main cluster. The shape of the decision-making process emerged.

The key, it turned out, was the brilliant way the bees exploited their diversity of knowledge—the second major principle of a smart swarm. Just as Deborah Gordon’s ant colonies used self-organization to adjust to changes in the environment, so the honeybees used diversity of knowledge to make good decisions. By diversity of knowledge, in this case, I mean a broad sampling of the swarm’s options. The more choices, the better. By sending out hundreds of scouts at the same time, each swarm collected a wealth of information about the neighborhood and the nest boxes, and it did so in a distributed and decentralized way. None of the bees tried to visit all five of the boxes to rate which one was the best. Nor did they submit their findings to some executive committee for a final decision, as workers in a corporation might do. Instead, these hundreds of scouts each provided unique information about the various sites to the group as a whole in what Seeley and Visscher described as a “friendly competition of ideas.”

Equally important, every scout evaluated nest sites for herself. If a scout was impressed by another scout’s dance, she might fly to the box being advertised and conduct her own inspection, which could last as long as an hour. But she would never blindly follow another scout’s opinion by dancing for a site she hadn’t visited. That would open the door to untested information being spread like a rumor. Or, to use the stock-broker analogy, a bee wouldn’t invest in a company just because its stock was on the rise. She’d check out its fundamentals first.

Meanwhile, as the scout bees continued their search, the swarm was busily ranking each option. This was determined by the number of bees visiting each site. The more visitors, the more “votes” for the site. Though the best nest box wasn’t discovered first by the scouts, it quickly attracted the attention of numerous bees. Scouts returning from the excellent box had no trouble convincing others to check it out, largely because they danced for it so vigorously—performing as many as a hundred dance circuits each, compared to only a dozen or so danced by bees for lesser sites. A dance of that length could take five minutes, compared with thirty seconds for a shorter dance, so it was much more likely to be noticed by scouts walking around on the surface of the cluster. And once the number of bees advertising the best box increased, support for it shot up, as interest in the mediocre sites faded away.

“This careful tuning of dance strength by the scouts created a powerful positive feedback,” Seeley explained, “which caused support for the best site to snowball exponentially.” This was a crucial mechanism, because it meant that even small differences in the quality of nests were exaggerated—their “signals” were amplified—making it much more likely that support for the best site would surge ahead.

As more and more bees gathered at the first-rate box, fewer and fewer lingered at the others. That was because scouts returning from boxes for the second or third time were dancing fewer circuits for them each time, whether they’d visited the excellent box or the mediocre ones. Scouts that had visited poor sites quit dancing first. Seeley and Visscher described this mechanism as the dance “decay rate.” It meant that support for less attractive boxes would dwindle automatically—even as the number of bees collecting at the superior box kept growing—in a decision-making process that lasted from two to five hours during the test. In technical terms, this represented a balancing, or negative feedback, preventing the swarm from choosing too fast and making a mistake. These were the factors steering the bees’ problem-solving machine—exponential recruitment on the accelerator, dance decay rate on the brakes.

Meanwhile, something critical was happening at the nest boxes. As soon as the number of bees visible near the entrance to the best box reached fifteen or so, Seeley and Visscher noticed a new behavior among the scouts. Those returning from the box started plowing through bees in the main cluster, producing a special signal called “worker piping.”

“It sounds like nnneeeep, nnneeeep! Like a race car revving up its engine,” Seeley says. “It’s a signal that a decision has been reached and it’s time for the rest of the swarm to warm up their wing muscles and prepare to fly.” Scouts from the excellent box, in other words, were announcing that a quorum had been reached. Enough bees had “voted” for the most attractive box by gathering there at the same time. A new home had been chosen.

The number fifteen, it turns out, was the threshold level for the quorum. Although this number might seem arbitrary at first glance, it turns out to be anything but that. Like the dance decay rate, the threshold level represents a finely tuned mechanism of emergence. To gather that many bees at the entranceway simultaneously, it takes as many as 150 scouts traveling back and forth between the box and the main swarm cluster, which means that a majority of the bees taking part in the selection process have committed themselves to the site.

Once the quorum was reached, the final step was for scouts to lead the rest of the group to the chosen site. Most of the swarm, some 95 to 97 percent, had been resting during the whole decision-making process, conserving their energy for the work ahead. Now, as the scouts scrambled through the crowd, they stopped from time to time to press their thoraxes against other bees to vibrate their wing muscles, as if to say, warm up, warm up, get ready to fly. A final signal, called the buzz run, in which the scouts bulldoze through sleepy workers and buzz their wings dramatically, triggered the takeoff. At that point, the whole swarm flew away to its new home—which, to nobody’s surprise, turned out to be the best nest box.

The swarm chose successfully, in short, because it made the most of its diversity of knowledge. By tapping into the unique information collected by hundreds of scouts, it maximized its chances of finding the best solution. By setting the threshold level high enough to produce a good decision, it minimized its chances of making a big mistake. And it did both in a timely manner under great pressure to be accurate.

The swarm worked so efficiently, in fact, you might be tempted to imagine it as a complicated Swiss watch, with hundreds of tiny parts, each one smoothly performing its function. Yet the reality is much more interesting. To watch a swarm in the midst of deliberation is to witness a chaotic scene not unlike the floor of a commodities market, with dozens of brokers shouting out orders at the same time. Bees coming and going. Scouts dancing this way or that. Uncommitted bees milling around. The way they make decisions looks very messy, which is also very beelike. Natural selection has fashioned a system that is not only tailor-made for their extraordinary talents for cooperation and communication but also forgiving of their tendency to be unpredictable. It is from this controlled messiness that the wisdom of the hive emerges.

Seek a diversity of knowledge. Encourage a friendly competition of ideas. Use an effective mechanism to narrow your choices. These are the lessons of the swarm’s success. They also happen be the same rules that enable certain groups of people to make smart decisions together—from antiterrorism teams to engineers in aircraft factories—through a surprising phenomenon that has come to be known as the “wisdom of crowds.”

The Wisdom of Crowds

In early 2005, Jeff Severts, a vice president at Best Buy, decided to try something different. Severts had recently attended a talk by James Surowiecki, whose bestseller The Wisdom of Crowds claims that, under the right circumstances, groups of nonexperts can be remarkably insightful. In some cases, Surowiecki argues, they can be even more intelligent than the most intelligent people in their ranks. Severts wondered if he might be able to tap into such braininess at Best Buy. As an experiment, in late January 2005 he sent e-mails to several hundred employees throughout the company, asking them to predict sales of gift cards in February. He got 192 replies. In early March, he compared the average of these estimates to actual sales for the month. The collective estimate turned out to be 99.5 percent accurate—almost 5 percent better than the figure produced by the team responsible for sales forecasts.

“I was surprised at how eerily accurate the crowd’s estimates were,” Severts says.

In his book about smart crowds, Surowiecki cites similar examples of otherwise ordinary people making extraordinary decisions. Take the quiz show Who Wants to Be a Millionaire? Contestants stumped by a question are given the option of telephoning an expert friend for advice or of polling the studio audience, whose votes are averaged by a computer. “Everything we know about intelligence suggests that the smart individual would offer the most help,” Surowiecki writes. “And in fact the ‘experts’ did okay, offering the right answer—under pressure—almost 65 percent of the time. But they paled in comparison to the audiences. Those random crowds of people with nothing better to do on a weekday afternoon than sit in a TV studio picked the right answer 91 percent of the time.”

Although Surowiecki readily admits that such stories by themselves don’t amount to scientific proof, they do raise a good question: If hundreds of bees can make reliable decisions together, why should it be so surprising that groups of people can too? “Most of us, whether as voters or investors or consumers or managers, believe that valuable knowledge is concentrated in a very few hands (or, rather, in a very few heads). We assume that the key to solving problems or making good decisions is finding that one right person who will have the answer,” Surowiecki writes. But often that’s a big mistake. “We should stop hunting and ask the crowd (which, of course, includes the geniuses as well as everyone else) instead. Chances are, it knows.”

Severts was so impressed by his first few efforts to harness collective wisdom at Best Buy that he and his team began experimenting with something called prediction markets, which represent a more sophisticated way of gathering forecasts about company performance from employees. In a prediction market, an employee uses play money to bid on the outcome of a question, such as “Will our first store in China open on time?” A correct bid pays $100, an incorrect bid pays nothing. If the current price of a share in the market for a bid that yes, the store will open on time, is $80, for example, that means the entire group believes there’s an 80 percent chance that that will happen. If an employee is more optimistic, believing there’s a 95 percent chance, he might take the bet, seeing an opportunity to earn $15 per share. In the case of the new store, which had been scheduled to open in Shanghai in December 2006, the prediction market took a dive, falling from $80 a share to $50 eight weeks before the opening date—even though official company forecasts at the time were still positive. In the end, the store opened a month late.

“That first drop was an early warning signal,” Severts says. “Some piece of new information came into the market that caused the traders to radically change their expectations.” What that new information might have been about, Severts never found out. But to him it didn’t really matter. The prediction market had proved its ability to overcome the many barriers to effective communication in a large company. If anyone was listening, the alarm bells were ringing loud and clear.

As this story suggests, there may be several good reasons for companies to pay attention to prediction markets, which are good at pulling together information that may be widely scattered throughout a corporation. For one thing, they’re likely to provide unbiased outlooks. Since bids are placed anonymously, markets may reflect the true opinions of employees, rather than what their bosses want them to say. For another thing, they tend to be relatively accurate, since the incentives for bidders to be correct—from T-shirts to cash prizes—encourage them to get it right, using whatever unique resources they might have.

Above and beyond these factors is the powerful way prediction markets leverage the simple mathematics of diversity ofknowledge, which, when applied with a little care, can turn a crowd of otherwise unremarkable individuals into a comparative genius. “If you ask a large enough group of diverse, independent people to make a prediction or estimate a probability, and then average those estimates, the errors each of them makes in coming up with an answer will cancel themselves out,” Surowiecki explains. “Each person’s guess, you might say, has two components: information and error. Subtract the error, and you’re left with the information.”

The house-hunting bees demonstrate this math very clearly. When several scouts return to the swarm from checking out the same perfect tree hollow, for example, they frequently give it different scores—like opinionated judges at an Olympic ice-skating competition. One bee might show great enthusiasm for such a high-quality site, dancing fifty waggle runs for it. Another might dance only thirty runs for it, while a third might dance only ten, even though she, too, approves of the site.

Scouts returning from a less attractive site, meanwhile, like a hole in a stone wall, might be reporting their scores on the swarm cluster at the same time, and they could show just as much variation. Let’s say these three bees dance forty-five runs, twenty-five runs, and five runs, respectively, in support of this mediumquality site. “You might think, gosh, this thing looks like a mess. Why are they doing it this way?” Tom Seeley says. “If you were relying on just one bee reporting on each site, you’d have a real problem, because one of the bees that visited the excellent site danced only ten runs, while one of the bees that visited the medium site did forty-five.” That could easily mislead you.

Fortunately for the bees, their decision-making process, like that of Olympics, doesn’t rely on the opinion of any single individual. Just as the scores given by the international judging committee are averaged after each skater’s performance, so the bees combine their assessments through competitive recruitment. “At the individual level, it looks very noisy, but if you say, well, what’s the total strength of all the bees from the excellent site, then the problem disappears,” Seeley explained. Add the three scores for the tree hollow—fifty, thirty, and ten—and you get a total of ninety waggle runs. Add the scores for the hole in the wall—forty-five, twenty-five, and five—and you get seventy-five runs. That’s a difference of fifteen runs, or 20 percent, between the two sites, which is more than enough for the swarm to choose wisely.

“The analogy is really quite powerful,” Surowiecki says. “The bees are predicting which nest site will be best, and humans can do the same thing, even in the face of exceptionally complex decisions.”

The key to such calculations, as we saw earlier, is the diversity of knowledge that individuals bring to the table, whether they’re scout bees, astronauts, or members of a corporate board. The more diversity the better—meaning the more strategies for approaching problems, the better; the more sources of information about the likelihood of something taking place, the better. In fact, Scott Page, an economist at the University of Michigan, has demonstrated that, when it comes to groups solving problems or making predictions, being different is every bit as important as being smart.

“Ability and diversity enter the equation equally,” he states in his book, The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies. “This result is not a political statement but a mathematical one, like the Pythagorean Theorem.”

By diversity, Page means the many differences we each have in the way we approach the world—how we interpret situations and the tools we use to solve problems. Some of these differences come from our education and experience. Others come from our personal identity, such as our gender, age, cultural heritage, or race. But primarily he’s interested in our cognitive diversity—differences in the problem-solving tools we carry around in our heads. When a group is struggling with a difficult problem, it helps if each member brings a different mix of tools to the job. That’s why, increasingly, scientists collaborate on interdisciplinary teams, and why companies seek out bright employees who haven’t all graduated from the same schools. “When people see a problem the same way, they’re likely all to get stuck at the same solutions,” Page writes. But when people with diverse problem-solving skills put their heads together, they often outperform groups of the smartest individuals. Diversity, in short, trumps ability.