The Drunkard's Walk: How Randomness Rules Our Lives

Critically analyzes the role of chance and random events, forces, and factors in shaping human existence, in a readable study of how the mathematical laws of randomness control the world around us. By the author of Feynman's Rainbow. 30,000 first printing.

Peering through the Eyepiece of Randomness  

I remember, as a teenager, watching the yellow flame of the Sabbath candles dancing randomly above the white paraffin cylinders that fueled them. I was too young to think candlelight romantic, but still I found it magical-because of the flickering images created by the fire. They shifted and morphed, grew and waned, all without apparent cause or plan. Surely, I believed, there must be rhyme and reason underlying the flame, some pattern that scientists could predict and explain with their mathematical equations. "Life isn't like that," my father told me. "Sometimes things happen that cannot be foreseen." He told me of the time when, in Buchenwald, the Nazi concentration camp in which he was imprisoned and starving, he stole a loaf of bread from the bakery. The baker had the Gestapo gather everyone who might have committed the crime and line the suspects up. "Who stole the bread?" the baker asked. When no one answered, he told the guards to shoot the suspects one by one until either they were all dead or someone confessed. My father stepped forward to spare the others. He did not try to paint himself in a heroic light but told me that he did it because he expected to be shot either way. Instead of having him killed, though, the baker gave my father a plum job, as his assistant. "A chance event," my father said. "It had nothing to do with you, but had it happened differently, you would never have been born." It struck me then that I have Hitler to thank for my existence, for the Germans had killed my father's wife and two young children, erasing his prior life. And so were it not for the war, my father would never have emigrated to New York, never have met my mother, also a refugee, and never have produced me and my two brothers.  

My father rarely spoke of the war. I didn't realize it then, but years later it dawned on me that whenever he shared his ordeals, it was not so much because he wanted me to know of his experiences but rather because he wanted to impart a larger lesson about life. War is an extreme circumstance, but the role of chance in our lives is not predicated on extremes. The outline of our lives, like the candle's flame, is continuously coaxed in new directions by a variety of random events that, along with our responses to them, determine our fate. As a result, life is both hard to predict and hard to interpret. Just as, looking at a Rorschach blot, you might see Madonna and I, a duck-billed platypus, the data we encounter in business, law, medicine, sports, the media, or your child's third-grade report card can be read in many ways. Yet interpreting the role of chance in an event is not like intepreting a Rorschach blot; there are right ways and wrong ways to do it.  

We often employ intuitive processes when we make assessments and choices in uncertain situations. Those processes no doubt carried an evolutionary advantage when we had to decide whether a saber-toothed tiger was smiling because it was fat and happy or because it was famished and saw us as its next meal. But the modern world has a different balance, and today those intuitive processes come with drawbacks. When we use our habitual ways of thinking to deal with today's tigers, we can be led to decisions that are less than optimal or even incongruous. That conclusion comes as no surprise to those who study how the brain processes uncertainty: many studies point to a close connection between the parts of our brain that make assessments of chance situations and those that handle the human characteristic that is often considered our prime source of irrationality-our emotions. Functional magnetic resonance imaging, for example, shows that risk and reward are assessed by parts of the dopaminergic system, a brain-reward circuit important for motivational and emotional processes. The images show, too, that the amygdala, which is also linked to our emotional state, especially fear, is activated when we make decisions couched in uncertainty.  

The mechanisms by which people analyze situations involving chance are an intricate product of evolutionary factors, brain structure, personal experience, knowledge, and emotion. In fact, the human response to uncertainty is so complex that sometimes different structures within the brain come to different conclusions and apparently fight it out to determine which one will dominate. For example, if your face swells to five times its normal size three out of every four times you eat shrimp, the "logical" left hemisphere of your brain will attempt to find a pattern. The "intuitive" right hemisphere of your brain, on the other hand, will simply say "avoid shrimp." At least that's what researchers found in less painful experimental setups. The game is called probability guessing. In lieu of toying with shrimp and histamine, subjects are shown a series of cards or lights, which can have two colors, say green and red. Things are arranged so that the colors will appear with different probabilities but otherwise without a pattern. For example, red might appear twice as often as green in a sequence like red-red-green-red-green-red-red-green-green-red-red-red, and so on. The task of the subject, after watching for a while, is to predict whether each new member of the sequence will be red or green.  

The game has two basic strategies. One is to always guess the color that you notice occurs more frequently. That is the route favored by rats and other nonhuman animals. If you employ this strategy, you are guaranteed a certain degree of success but you are also conceding that you will do no better. For instance, if green shows up 75 percent of the time and you decide to always guess green, you will be correct 75 percent of the time. The other strategy is to "match" your proportion of green and red guesses to the proportion of green and red you observed in the past. If the greens and reds appear in a pattern and you can figure out the pattern, this strategy enables you to guess right every time. But if the colors appear at random, you would be better off sticking with the first strategy. In the case where green randomly appears 75 percent of the time, the second strategy will lead to the correct guess only about 6 times in 10.  

Humans usually try to guess the pattern, and in the process we allow ourselves to be outperformed by a rat. But there are people with certain types of post-surgical brain impairment-called a split brain-that precludes the right and left hemispheres of the brain from communicating with each other. If the probability experiment is performed on these patients such that they see the colored light or card with only their left eye and employ only their left hand to signal their predictions, it amounts to an experiment on the right side of the brain. But if the experiment is performed so as to involve only their right eye and right hand, it is an experiment on the left brain. When researchers performed those experiments, they found that-in the same patients-the right hemisphere always chose to guess the more frequent color and the left hemisphere always tried to guess the pattern.  

Making wise assessments and choices in the face of uncertainty is a rare skill. But like any skill, it can be improved with experience. In the pages that follow, I will examine the role of chance in the world around us, the ideas that have been developed over the centuries to help us understand that role, and the factors that often lead us astray. The British philosopher and mathematician Bertrand Russell wrote,  

We all start from "naive realism," i.e., the doctrine that things are what they seem. We think that grass is green, that stones are hard, and that snow is cold. But physics assures us that the greenness of grass, the hardness of stones, and the coldness of snow are not the greenness of grass, the hardness of stones, and the coldness of snow that we know in our own experience, but something very different. In what follows we will peer at life through the eyepiece of randomness and see that many of the events of our lives, too, are not quite what they seem but rather something very different.  
In 2002 the Nobel committee awarded the Nobel Prize in Economics to a scientist named Daniel Kahneman. Economists do all sorts of things these days-they explain why teachers are paid so little, why football teams are worth so much, and why bodily functions help set a limit on the size of hog farms (a hog excretes three to five times as much as a human, so a farm with thousands of hogs on it often produces more waste than the neighboring cities). Despite all the great research generated by economists, the 2002 Nobel Prize was notable because Kahneman is not an economist. He is a psychologist, and for decades, with the late Amos Tversky, Kahneman studied and clarified the kinds of misperceptions of randomness that fuel many of the common fallacies I will talk about in this book.  

The greatest challenge in understanding the role of randomness in life is that although the basic principles of randomness arise from everyday logic, many of the consequences that follow from those principles prove counterintuitive. Kahneman and Tversky's studies were themselves spurred by a random event. In the mid-1960s, Kahneman, then a junior psychology professor at Hebrew University, agreed to perform a rather unexciting chore: lecturing to a group of Israeli air force flight instructors on the conventional wisdom of behavior modification and its application to the psychology of flight training. Kahneman drove home the point that rewarding positive behavior works but punishing mistakes does not. One of his students interrupted, voicing an opinion that would lead Kahneman to an epiphany and guide his research for decades.  

"I've often praised people warmly for beautifully executed maneuvers, and the next time they always do worse," the flight instructor said. "And I've screamed at people for badly executed maneuvers, and by and large the next time they improve. Don't tell me that reward works and punishment doesn't work. My experience contradicts it." The other flight instructors agreed. To Kahneman the flight instructors' experiences rang true. On the other hand, Kahneman believed in the animal experiments that demonstrated that reward works better than punishment. He ruminated on this apparent paradox. And then it struck him: the screaming preceded the improvement, but contrary to appearances it did not cause it.  

How can that be? The answer lies in a phenomenon called regression toward the mean. That is, in any series of random events an extraordinary event is most likely to be followed, due purely to chance, by a more ordinary one. Here is how it works: The student pilots all had a certain personal ability to fly fighter planes. Raising their skill level involved many factors and required extensive practice, so although their skill was slowly improving through flight training, the change wouldn't be noticeable from one maneuver to the next. Any especially good or especially poor performance was thus mostly a matter of luck. So if a pilot made an exceptionally good landing-one far above his normal level of performance-then the odds would be good that he would perform closer to his norm-that is, worse-the next day. And if his instructor had praised him, it would appear that the praise had done no good. But if a pilot made an exceptionally bad landing-running the plane off the end of the runway and into the vat of corn chowder in the base cafeteria-then the odds would be good that the next day he would perform closer to his norm-that is, better. And if his instructor had a habit of screaming "you clumsy ape" when a student performed poorly, it would appear that his criticism did some good. In this way an apparent pattern would emerge: student performs well, praise does no good; student performs poorly, instructor compares student to lower primate at high volume, student improves. The instructors in Kahneman's class had concluded from such experiences that their screaming was a powerful educational tool. In reality it made no difference at all.  

This error in intuition spurred Kahneman's thinking. He wondered, are such misconceptions universal? Do we, like the flight instructors, believe that harsh criticism improves our children's behavior or our employees' performance? Do we make other misjudgments when faced with uncertainty? Kahneman knew that human beings, by necessity, employ certain strategies to reduce the complexity of tasks of judgment and that intuition about probabilities plays an important part in that process. Will you feel sick after eating that luscious-looking seviche tostada from the street vendor? You don't consciously recall all the comparable food stands you've patronized, count the number of times you've spent the following night guzzling Pepto-Bismol, and come up with a numerical estimate. You let your intuition do the work. But research in the 1950s and early '60s indicated that people's intuition about randomness fails them in such situations. How widespread, Kahneman wondered, was this misunderstanding of uncertainty? And what are its implications for human decision making? A few years passed, and Kahneman invited a fellow junior professor, Amos Tversky, to give a guest lecture at one of his seminars. Later, at lunch, Kahneman mentioned his developing ideas to Tversky. Over the next thirty years, Tversky and Kahneman found that even among sophisticated subjects, when it came to random processes-whether in military or sports situations, business quandaries, or medical questions-people's beliefs and intuition very often let them down.  

Suppose four publishers have rejected the manuscript for your thriller about love, war, and global warming. Your intuition and the bad feeling in the pit of your stomach might say that the rejections by all those publishing experts mean your manuscript is no good. But is your intuition correct? Is your novel unsellable? We all know from experience that if several tosses of a coin come up heads, it doesn't mean we are tossing a two-headed coin. Could it be that publishing success is so unpredictable that even if our novel is destined for the best-seller list, numerous publishers could miss the point and send those letters that say thanks but no thanks? One book in the 1950s was rejected by publishers, who responded with such comments as "very dull," "a dreary record of typical family bickering, petty annoyances and adolescent emotions," and "even if the work had come to light five years ago, when the subject [World War II] was timely, I don't see that there would have been a chance for it." That book, The Diary of a Young Girl by Anne Frank, has sold 30 million copies, making it one of the best-selling books in history. Rejection letters were also sent to Sylvia Plath because "there certainly isn't enough genuine talent for us to take notice," to George Orwell for Animal Farm because "it is impossible to sell animal stories in the U.S.," and to Isaac Bashevis Singer because "it's Poland and the rich Jews again." Before he hit it big, Tony Hillerman's agent dumped him, advising that he should "get rid of all that Indian stuff."