Don’t Trust Your Gut
2/3 Free Articles leftRemaining
Guest User
Loading...
Don’t Trust Your Gut
<stream-item class="stream-item"
data-id="tag:hbr.org,:40-R0305J"
data-title="Don’t Trust Your Gut"
data-url="/2003/05/dont-trust-your-gut"
data-topic="Decision making"
data-authors="Eric Bonabeau"
data-content-type="Digital Article"
data-content-image=""
data-summary="Intuition plays an important role in decision making, but it can be dangerously unreliable in complicated situations. A new set of analytical tools can help you leverage your instinct without being sabotaged by its weaknesses.
View more from the
Loading...
The Idea in Brief
Executives today are in a real bind: The choices facing us—and the data requiring our analysis—have multiplied, while time for decision-making has shrunk. One decision-making tool—intuition—seems to offer a reliable, faster alternative to painstaking data gathering and study. Indeed, 45% of executives rely more on instinct than facts and figures to run their businesses.
Intuition has its place in decision making. But detached from rigorous analysis, it’s fickle—leading to disaster as often as success. The more options you’re evaluating, and the more complex and unfamiliar your challenges, the less you should emphasize instinct over reason.
How to break the bind of too many choices, too little time? Use decision-support software to quickly sort through your many alternatives—and select the best ones.
The Idea in Practice
Our Untrustworthy Gut
Intuition—interpreting and reaching conclusions about phenomena without conscious thought—carries dangerous biases. We givedisproportionate weight to:
information confirming, not challenging,
conclusions justifying, not upending, and
information we receive first—which distorts our interpretation of subsequent data.
Most troubling, we seek patterns in new situations even when they don’t exist. Interpretinga new threat by squeezing it into an old pattern, we miss what makes it different—and risk taking the wrong action. Pattern seeking also cuts off thinking when we should stay open to choices as long as possible before making final decisions.
Expanding the Mind
If we can’t rely on intuition—but lack time to analyze complex situations—how can we make the best choices? Consider these computer-based tools:
helps managers make decisions about problems with many interrelated but unpredictable elements—such as global markets, supply chains, and large organizations. A computer creates thousands of individual actors, then each makes decisions, simulating complex systems’ dynamics. Southwest Airlines used agent-based modeling to revamp its cargo-handling rules—saving $2 million on labor annually.
In, computers emulate nature—combining and mutating the best available options to create even better ones.
You run a factory and want to determine the production schedule that will maximize plant output. You randomly generate alternative schedules and feed them into your software—which evaluates each schedule, picks the few best, then randomly pairs them. The resulting set combines the prior generation’s characteristics while introducing random characteristics as mutations. The software continues mating rounds, generating increasingly better solutions.
In, people judge each generation of computer-generated alternatives. One automobile manufacturer employed artificial-evolution software to pump out new design iterations quickly—then designers used subjective aesthetic judgment to pick promising ones for new mating rounds.
In, software generates and sorts through potential solutions to business problems with unbounded options and ill-defined success criteria.
A petrochemical company uses open-ended searching to evaluate innumerable pricing-strategy options. The software breaks strategies into component parts
for example, “If volume is & 100, price = X” and “If winter is cold, price decreases.” The computer adds random rules, then additional rule combinations, to produce new strategies for testing.
Making high-stakes business decisions has always been hard. But in recent decades, as the complexities of global commerce have deepened, it’s become tougher than ever. The choices facing managers and the data requiring analysis have multiplied even as the time for analyzing them has shrunk.
One decision-making tool—human intuition—seems to offer a reliable alternative to painstaking fact gathering and analysis. Encouraged by scientific research on intuition, top managers feel increasingly confident that, when faced with complicated choices, they can just trust their gut. Indeed, a survey that was conducted in May 2002 by executive search firm Christian & Timbers reveals that fully 45% of corporate executives now rely more on instinct than on facts and figures in running their businesses. Decision-making consultant Gary Klein, in his book Intuition at Work, expresses the common wisdom when he says that intuition is “at the center of the decision-making process” and that analysis is, at best, “a supporting tool for making intuitive decisions.”
The trust in intuition is understandable. People have always sought to put their faith in mystical forces when confronted with earthly confusion. But it’s also dangerous. Intuition has its place in decision making—you should not ignore your instincts any more than you should ignore your conscience—but anyone who thinks that intuition is a substitute for reason is indulging in a risky delusion. Detached from rigorous analysis, intuition is a fickle and undependable guide—it is as likely to lead to disaster as to success. And while some have argued that intuition becomes more valuable in highly complex and changeable environments, the opposite is actually true. The more options you have to evaluate, the more data you have to weigh, and the more unprecedented the challenges you face, the less you should rely on instinct and the more on reason and analysis.
That brings us back to the essential conundrum facing today’s harried executive: How do you analyze more in less time? The answer may lie, it now appears, in technology. Powerful new decision-support tools can help executives quickly sort through vast numbers of alternatives and pick the best ones. When combined with the experience, insight, and analytical skills of a good management team, these tools offer companies a way to make consistently sound and rational choices even in the face of bewildering complexity—a capability that intuition will never match.
Intuition’s Allure
The stories are certainly seductive. Fred Smith has an insight into the transport business and, despite widespread skepticism, goes on to create Federal Express. Michael Eisner hears a pitch for an offbeat game show and, knowing in his heart it’s going to be a blockbuster, immediately commits millions to developing Who Wants to Be a Millionaire? George Soros senses in his bones a big shift in currency markets and, acting on that hunch, makes a billion-dollar killing. Robert Pittman has a vision of the future of on-line media while taking a shower and rushes to lead America Online in an entirely new direction.
The reason such tales (whether apocryphal or not) have become business legends is that we want to believe in the transformative power of intuition. For one thing, it’s romantic. It raises business above the drab world of spreadsheets and income statements and turns it into something of an art form. The executive office becomes a place of inspiration and vision rather than planning and number crunching. For another, it simplifies. It says that we needn’t worry if we can’t decipher complex challenges rationally—our subconscious mind will automatically deliver the right answer. We just need to relax, close our eyes, and let the magic happen.
Finally, it makes us feel special. Any idiot can run the numbers, but the gift of a good gut—that’s reserved for the true business elite. Two years ago in these pages, Johnson & Johnson CEO Ralph Larsen gave voice to this common, if unproven, assumption: “Very often, people will do a brilliant job up through the middle management levels, where it’s very heavily quantitative in terms of the decision-making. But then they reach senior management, where the problems get more complex and ambiguous, and we discover that their judgment or intuition is not what it should be.” What better way to justify a high status—and a huge salary—than to claim the superhuman power of exceptional instinct.
But our desire to believe in the wisdom of intuition blinds us to the less romantic realities of business decision making. We remember the examples of hunches that pay off but conveniently forget all the ones that turn out badly. FedEx’s Fred Smith also launched ZapMail, a proprietary network for fax transmissions that bombed. Michael Eisner was responsible for the debacle of the EuroDisney opening, not to mention recent box-office turkeys The Country Bears and Treasure Planet. George Soros lost a fortune speculating in Russian securities in the late 1990s and then promptly lost another one betting on tech stocks in 2000. And as for AOL’s Pittman, his instinctive belief that the company’s future lay in advertising rather than subscriptions now appears to be less a brilliant insight than a brilliant mistake—and one of the reasons he’s no longer employed at AOL. The unhappy fact that we’d prefer not to admit to ourselves is this: For every example of a great gut decision, there’s an equal and opposite example of a terrible one.
Our Untrustworthy Gut
Critiques of intuition are complicated by the fact that “intuition” is such a slippery word. Its definition can be stretched to mean almost anything, from innate instinct to professional judgment to plain-old common sense. But people generally agree that intuition refers to the brain’s process of interpreting and reaching conclusions about phenomena without resorting to conscious thought. And further, it’s usually assumed that this process draws on the mind’s vast storehouse of memories. Bruce Henderson, founder of the Boston Consulting Group, may have put it best when, in 1977, he called intuition “the subconscious integration of all the experiences, conditioning, and knowledge of a lifetime, including the cultural and emotional biases of that lifetime.”
It’s certainly true that the mind is a marvelous processor of information—we would be lost in the world without its hidden stream of calculations. But it’s also true, as Henderson intimated, that it’s an imperfect processor. Scholars of human cognition have shown that our thinking is subject to all sorts of biases and flaws, most of which operate at a subconscious level—at the level, in other words, of intuition. We naturally give more weight to information that confirms our assumptions and prejudices, for example, while dismissing information that would call them into question. We’re also creatures of the status quo, drawn to conclusions that justify and perpetuate current conditions and repelled by anything that would roil the waters. And we’re irrationally influenced by the first information we receive on a particular subject—it becomes, as decision researchers put it, the “anchor” that determines and distorts how we process all subsequent data.
The most dangerous of these flaws, when it comes to intuition, is our deep-seated need to see patterns. The mind’s well-documented facility for pattern recognition seems to lie at the very core of intuition—it’s how the brain synthesizes information from the past and uses it to understand the present and anticipate the future. But it can get us into trouble. Researchers have shown that our unconscious desire to identify patterns is so strong that we routinely perceive them where they don’t in fact exist. When confronted with a new phenomenon, our brains try to categorize it based on our previous experiences, to fit it into one of the patterns stored in our memories. The problem is that, in making that fit, we inevitably filter out the very things that make the new phenomenon new—we rush to recycle the reactions and solutions from the past.
That instinct, seemingly hardwired into our thinking by evolution, is extremely useful in life-or-death situations where fine distinctions are irrelevant. If you were a caveman and had seen strange animals maul other cavemen in the past, then it would probably be wise for you to flee from any strange animal you happened to come across—even if you’d never seen the beast before. The benefit of a careful analysis of the situation would be far outweighed by the risk of inaction. But managers are not cavemen. In complex business situations, fine distinctions do matter—often, they’re precisely what separates success from failure. If you try to interpret a competitive threat or market upheaval by simply squeezing it into an old pattern, you’re likely to miss what makes it different—and take the wrong action. Intuition is a means not of assessing complexity but of ignoring it. That’s valuable if you’re a firefighter in a burning building or a soldier on a battlefield. It’s not valuable if you’re an executive faced with a pressing decision about investing millions in a new product for a rapidly changing market.
The more complex the situation, the more misleading intuition becomes. In a truly chaotic environment—where cause and effect no longer have a linear relationship—the last thing you want to do is try to apply patterns to it. The essence of such an environment is the lack of any discernible pattern in its evolution. In his McKinsey Quarterly article “On the Origin of Strategies,” consultant Eric Beinhocker put it this way: “The properties of complex adaptive systems present particular challenges to the development of business strategy because people have a natural tendency to look for patterns. Indeed, the human drive to find patterns is so strong that they are often read into perfectly random data. Moreover, human beings like to assume that cause directly precedes effect, which makes it difficult to anticipate the second-, third-, and fourth-order effects of path dependence.” If you make an intuitive decision that turns out well in such a situation, it’s because you’re lucky, not gifted. And sooner or later—probably sooner—your luck is going to run out. Just ask your average day trader.
The instinctive rush to apply a pattern to a phenomenon can also cut off or narrow an individual’s or a group’s thinking too quickly. Impatient with ambiguity, the mind naturally seeks closure—that seems, in fact, to be one of the main functions of intuition—but an intelligent decision-making process often requires the sustained exploration of many alternatives. You want to keep the process open as long as possible before converging on a final choice. That’s hard to do when your gut—or your boss’s gut—is giving you The Answer.
Intuition presents another, even more insidious problem: It masks me-too thinking. We like to assume that our intuition is uniquely our own, a distillation of our particular experiences and insights. But while that may have been true a century ago, when people led very different lives depending on where they lived and what they did, it’s no longer the case. In today’s global village, with its instantaneous and unceasing communications, human existence has become homogenized—we share the same experiences, the same opinions, even the same thoughts. We live in a vast echo chamber, and the voice of intuition we hear inside our heads is increasingly the same voice that speaks to everyone else. If, in making business decisions, we blindly follow its counsel, we’ll end up mimicking our competitors rather than creating strategies that distinguish us and bring us profits.
Expanding the Mind
So, if we can’t rely on our intuition but have neither the time nor the mental capacity to carefully analyze all the facets of a complex situation, how in the world can we make smart choices? Technology may hold the key. Sophisticated computer programs are now being developed that can supplement and bolster people’s decision-making skills. Many of these new decision-support tools are still in the early stages of development and have yet to be applied to strategic business decisions. But they hold enormous potential for helping executives carry out the two key components of decision-making or problem-solving exercises: searching for possible solutions and evaluating those solutions in order to choose the best one or ones. The more complex and fast-changing the situation, the more challenging both search and evaluation become. By expanding the analytical as well as the intuitive capabilities of the mind, the new programs allow a much faster, a much fuller, and a much more rigorous exploration of the options. (See the sidebar “Search and Evaluate” for an overview of traditional and emerging decision-support tools.)
The more complex the search or the evaluation, the more difficult it becomes for a person to carry it out – the required computations outstrip the mind’s processing capabilities. In such cases, some people will mistakenly rely on their intuition to si they’ll narrow their options or make a choice based on their gut. But intuition is particularly unreliable in complex situations. A much better approach, when you’re faced with a complex search or evaluation, is to supplement the mind’s analytical and intuitive capabilities with a computational decision-support tool.
The large figure categorizes both traditional and emerging decision-support methods and tools in terms of how they apply to different situations. There are many such tools, ranging from real options to visualization software, in common use today. Most traditional tools (indicated by gray type) have limited applicability in highl they’re best applied to problems that fall into or near the lower-left quadrant – those requiring relatively simple searches and evaluations. As we move outward on the complexity scale, we need to look to new, computer-based computational tools, such as open-ended search (when there are lots of potential solutions), agent-based modeling (when the consequences requiring evaluation are complex) or artificial evolution (when both search and evaluation are highly complex).
Decision Sciences.
The traditional tools of decision sciences—system dynamics, decision trees, real options, portfolio management, and so on—constitute an important class of rational decision-making techniques that can be invaluable when you’re faced with lots of options. They often lead to much more dependable decisions than does instinct alone. But they have their limits. Their workings are often so mysterious to executives that they can seem like black boxes. And in highly complex situations—when there are many dependencies among possible solutions or no clear way of measuring the solutions’ value—traditional tools become unwieldy and tend to provide unreliable answers.
To use decision trees in the pharmaceutical industry, for example, you have to assume you know a drug’s commercial value ten years before it hits the market. And decision trees and other decision-science tools can’t adequately account for emergent phenomena or chance events, such as the discovery that a drug developed for one disease can be used to treat another, very different disease.
Agent-Based Modeling.
Isaac Newton, after losing his savings in the South Sea Bubble of 1720, bemoaned the fact that “I can calculate the motions of the heavenly bodies, but not the madness of people.” Many managers today are in the same quandary as Newton was almost 300 years ago. They have to make decisions about complex systems with many interrelated, yet unpredictable, elements. Global markets, large organizations, supply chains, technology networks—all can seem impenetrable to traditional forms of analysis.
But agent-based modeling can shed light on the workings and evolution of such systems. In an agent-based simulation, a computer creates thousands, even millions,
each of these virtual agents makes decisions, providing an accurate model of a complex system’s dynamics. Agent-based modeling allows you, literally, to do what Newton couldn’t: predict the madness of crowds. (For more on agent-based modeling, see my HBR article “Predicting the Unpredictable,” March 2002.)
The best system ever devised for making choices from an almost infinite set of alternatives is evolution itself.
Southwest Airlines is using an agent-based model to revamp its rules for handling cargo, reaping $2 million in annual labor savings in the process. Eli Lilly is using one to model early-phase drug development, leading to the creation of organizational forms that promise to boost productivity and enhance speed. Pacific Gas and Electric is using an agent-based model to better manage the flow of electrons through its vast power grid, saving money and avoiding service disruptions.
In the coming years, agent-based models will no doubt be used to generate scenarios for the evolution of markets and competition, the dynamics of which hinge on the decisions made by many players. These scenarios can become the basis for evaluating a multitude of strategic and tactical options—and they can be used to put executives’ intuitive choices to the test.
Artificial Evolution.
The best system ever devised for making choices from an almost infinite set of alternatives is evolution itself. The basic process of evolution—taking the best-available options and then combining and mutating them to create even better ones—is now being incorporated into a type of analytical software known as artificial evolution, or evolutionary computation. This technology uses the computational power of computers to both search out a vast number of solutions and evaluate them.
To see how it works, imagine that you run a factory and have to determine the production schedule that will maximize the plant’s output within a given period. You start by randomly generating some alternative schedules—their quality makes no difference at this point—and feeding them into artificial-evolution software. The software evaluates how well each schedule performs in maximizing output, picks the few that perform best, and randomly pairs them for “mating.” The resulting large set of alternative schedules combines the characteristics of the prior generation while introducing some random characteristics as mutations. It searches out, in other words, a large new set of possible solutions. The software evaluates the solutions, and the ones that perform best in maximizing output are selected for another round of mating. As more and more generations go by—and computers can crank through the process in minutes—the resulting schedules become better and better. John Deere already uses this kind of system to help optimize its manufacturing operations, and Mexican cement producer Cemex uses a similar system to route its trucks.
Interactive Evolution.
In the plant-scheduling example, alternatives could be judged with an objective measure—factory output. As decisions become more strategic, however, the criteria for success become more complex and subjective. You can’t
you have to incorporate the expertise, judgment, and, yes, intuition of seasoned professionals. You have to bring people into the evaluation stage of the decision-making process. That can be accomplished with interactive evolution, a variation of artificial evolution. The basic difference is that a person or group of people, rather than a computer, judges each generation of alternatives.
One major automobile manufacturer is using interactive evolution to aid in new-car design. That process is highly complex because car designers have to satisfy hundreds of technical constraints, such as wheelbase length, windshield angle, and engine compartment size, while also being creative in both engineering and aesthetics. When designers have to do this without the help of technology, it is extraordinarily time consuming. They have to test every decision against all sorts of variables, and as a result they can consider only a small set of options. But interactive-evolution software can pump out iterations of new designs very quickly. The designers examine each set of alternatives and, using subjective aesthetic judgments in addition to the computer’s objective measures, choose the best ones for the next round of mating.
Other companies, like Procter & Gamble and Pepsi-Cola North America, are using interactive evolution to create new product and packaging designs—but they’re using customers rather than employees to pick out the best options from each generation. One can easily imagine a similar process for high-level strategic decisions that leverages the insights of an executive team to continuously refine plans.
Open-Ended Search.
Artificial and interactive evolution are both optimization processes. Alternative designs are generated by varying a small set of parameters, and those designs are evaluated against a set of criteria—objective, subjective, or both. But sometimes you don’t know which parameters to use to generate alternatives, or the number of parameters is so large that it’s impossible to reliably sample the entire set of possible solutions. In such cases, another new computational technique—open-ended search, or evolutionary design—can be applied to sort through and to generate options. As its name implies, open-ended search focuses on the initial search for options rather than on their subsequent evaluation. It has enormous potential for helping managers make decisions in highly complex situations because it offers a way to generate options that would be invisible to even the most capacious mind.
Stanford professor John Koza has developed a type of open-ended search, called genetic programming, for use in creating electronic circuits. The number of possible circuits is huge, and it’s impossible to characterize all of them with just a few parameters. Using a small number of parameters (which is all the mind can handle) restricts the search to a tiny, predefined subset of circuits, precluding truly creative solutions from emerging. Genetic programming, by contrast, “dis-integrates” circuits into their component building blocks—diodes, amplifiers, resistors, and so forth—then uses a computer to breed alternative circuits by combining and recombining the components.
The process has generated radically new designs—ones that would never have been discovered by simply judging complete circuits against traditional performance criteria. Koza and his colleagues at Genetic Programming in Los Altos, California, have recently been using the technique to create circuits that replicate the functionality of other circuits without infringing on existing patents—a development that could, for better or worse, revolutionize the microchip industry.
My firm, Icosystem, has begun helping a major petrochemical company use open-ended search to evaluate pricing strategies for one of its most important products. The product’s pricing has to take into account many factors. These include upstream commodity prices, downstream finished-product prices, demand at various stages in the value chain, currency fluctuations, and competitor prices, all of which can change rapidly and unpredictably. As with the electronic-circuit example, the open-ended design begins with the disaggregation of an initial group of pricing strategies (which the company collects from various pricing experts) into their component parts. In this particular case, the parts take the form of pricing rules, as follows: “If volume is & 100, then price = x,” or, “If winter is cold, price decreases.”
To this primordial soup are added random rules—some of which directly contradict the experts’ rules—to add greater genetic diversity to the mix. A computer creates random combinations of the rules to produce a new set of strategies for testing. In this way, the computer can quickly explore millions of combinations, producing innovative strategies that go well beyond anything that might have come out of the conscious or subconscious minds of even the savviest marketers. And, again, it’s easy to see how open-ended search could be applied to complex strategic challenges that have many possible solutions. Just as with interactive evolution, people can aid in the evaluation of the options generated by open-ended search. The technique offers a rational way for managers to approach the most difficult business problems: those that have unbounded options with no well-defined criteria for success.
Beyond Intuition
These new decision-support tools don’t elimi they harness its power while remedying its most pernicious flaws. The instincts of smart executives and other professionals are incorporated into the process—they’re used either to generate initial options or to aid in judging computer-generated ones. But these instincts are subjected to the rigors of analysis and at the same time freed from the brain’s constraints in imagining possible solutions. Computers impose left-brain discipline on right-brain hunches—in a way that’s well beyond the computational capacity of the human mind. Intuition is thus allowed to inform decision making without short-circuiting or otherwise constraining it.
But there’s more to it than that. Ultimately, computers may not just amplify the mind’s ana they may expand its creative potential as well. And they may allow us to break through the interpretation barrier—our demand that our creations be intelligible to us.
Think about it. When we create designs, whether for products or strategies, we are limited by our ability to understand those designs—their workings must be transparent to us. But if we look at nature, we quickly find that some of its greatest creations are opaque—they lie beyond our understanding. That’s true of the human mind itself, perhaps the greatest creation of all. We don’ we just know that it works extraordinarily well. Techniques like artificial evolution and open-ended design can also generate designs that we can’t explain but that produce results beyond even the limits of our imaginations. They offer, it might be said, the true fulfillment of the promise of human intuition.
A version of this article appeared in the
issue of Harvard Business Review.
Eric Bonabeau () is the chairman of Icosystem, a strategy-consulting firm in Cambridge, Massachusetts.
Loading...
Loading...
Partner Center
U.S./Canada:
International:
U.S./Canada:
International: