Leadership and the New Science: Discovering Order in a Chaotic World (Revised and Expanded 2nd Edition)

Book by Wheatley Margaret J

To my mind there must be, at the bottom of it all, not an equation, but an utterly simple idea. And to me that idea, when we finally discover it, will be so compelling, so inevitable, that we will say to one another, “Oh, how beautiful. How could it have been otherwise?”

—John Archibald Wheeler
Introduction
Searching for a Simpler Way to Lead Organizations

I am not alone in wondering why organizations aren’t working well. Many of us are troubled by questions that haunt our work. Why do so many organizations feel lifeless? Why do projects take so long, develop evergreater complexity, yet too often fail to achieve any truly significant results? Why does progress, when it appears, so often come from unexpected places, or as a result of surprises or synchronistic events that our planning had not considered? Why does change itself, that event we’re all supposed to be “managing,” keep drowning us, relentlessly making us feel less capable and more confused? And why have our expectations for success diminished to the point that often the best we hope for is endurance and patience to survive the frequent disruptive forces in our organizations and lives?

These questions had been growing within me for several years, gnawing away at my work and diminishing my sense of competency. The busier I became with work and the more projects I took on, the greater my questions grew. Until I began a journey.

Like most important journeys, mine began in a mundane place—a Boeing 757, flying soundlessly above America. High in the air as a weekly commuter between Boston and Salt Lake City, with long stretches of reading time broken only by occasional offers of soda and peanuts, I opened my first book on the new science—Fritjof Capra’s The Turning Point, which describes the new world view emerging from quantum physics. This provided my first glimpse of a new way of perceiving the world, one that comprehended its processes of change, its deeply patterned nature, and its dense webs of connections.

I don’t think it accidental that I was introduced to a new way of seeing at 37,000 feet. The altitude only reinforced the message that what was needed was a larger perspective, one that took in more of the whole of things. From that first book, I took off, reading as many new science books as I could find in biology, evolution, chaos theory, and quantum physics. Discoveries and theories of new science called me away from the details of my own field of management and raised me up to a vision of the inherent orderliness of the universe, of creative processes and dynamic, continuous change that still maintained order. This was a world where order and change, autonomy and control were not the great opposites that we had thought them to be. It was a world where change and constant creation were ways of sustaining order and capacity.

I don’t believe I could have grasped these ideas if I had stayed on the ground.

During the past several decades, books that relate new science findings for lay readers have proliferated, some more reputable and scientific than others. Of the many I read, some were too challenging, some were too bizarre, but others contained images and information that were breathtaking. I became aware that I was wandering in a realm that created new visions of freedom and possibility, giving me new ways to think about my work. I couldn’t always draw immediate connections between science and my dilemmas, but I noticed myself developing a new serenity in response to the questions that surrounded me. I was reading of chaos that contained order; of information as an essential, nourishing element; of systems that fell apart so they could reorganize themselves; and of invisible influences that permeate space and affect change at a distance. These were compelling, evocative ideas, and they gave me hope, even if they did not reveal immediate solutions.

Somewhere—I knew then and believe even more firmly now—there is a simpler way to lead organizations, one that requires less effort and produces less stress than our current practices. For me, this new knowledge is now crystallizing into applications even as I realize that this exploration will take many years. But I no longer believe that organizations are inherently unmanageable in this world of constant flux and unpredictability. Rather, I believe that our present ways of organizing are outmoded, and that the longer we remain entrenched in our old ways, the further we move from those wonderful breakthroughs in understanding that the world of science calls “elegant.” The layers of complexity, the sense of things being beyond our control and out of control, are but signals of our failure to understand a deeper reality of organizational life, and of life in general.

We are all searching for this simpler way. In every academic discipline and institution, we live today with questions for which our expertise provides no answers. At the turn of the century, physicists faced the same unnerving confusion. There is a frequently told story about Niels Bohr and Werner Heisenberg, two founders of quantum theory. This version is from The Turning Point:

In the twentieth century, physicists faced, for the first time, a serious challenge to their ability to understand the universe. Every time they asked nature a question in an atomic experiment, nature answered with a paradox, and the more they tried to clarify the situation, the sharper the paradoxes became. In their struggle to grasp this new reality, scientists became painfully aware that their basic concepts, their language, and their whole way of thinking were inadequate to describe atomic phenomena. Their problem was not only intellectual but involved an intense emotional and existential experience, as vividly described by Werner Heisenberg: “I remember discussions with Bohr which went through many hours till very late at night and ended almost in despair; and when at the end of the discussion I went alone for a walk in the neighboring park I repeated to myself again and again the question: Can nature possibly be so absurd as it seemed to us in these atomic experiments?”

It took these physicists a long time to accept the fact that the paradoxes they encountered are an essential aspect of atomic physics. ... Once this was perceived, the physicists began to learn to ask the right questions and to avoid contradictions ... and finally they found the precise and consistent mathematical formulation of [quantum] theory.

. . . Even after the mathematical formulation of quantum theory was completed, its conceptual framework was by no means easy to accept. Its effect on the physicists’ view of reality was truly shattering. The new physics necessitated profound changes in concepts of space, time, matter, object, and cause and effect; and because these concepts are so fundamental to our way of experiencing the world, their transformation came as a great shock. To quote Heisenberg again: “The violent reaction to the recent development of modern physics can only be understood when one realizes that here the foundations of physics have started moving; and that this motion has caused the feeling that the ground would be cut from science.” (Capra 1983, 76–77)

For the past several years, I have found myself often relating this story to groups of people in organizations everywhere. The story speaks with a chilling familiarity. Each of us recognizes the feelings this tale describes, of being mired in the habit of solutions that once worked yet that are now totally inappropriate, of having rug after rug pulled from beneath us, whether by a corporate merger, reorganization, downsizing, or personal disorientation. But the story also gives great hope as a parable teaching us to embrace our despair as a step on the road to wisdom, encouraging us to sit in the unfamiliar seat of not knowing and open ourselves to radically new ideas. If we bear the confusion, then one day, the story promises, we will begin to see a whole new land, one of bright illumination that will dispel the oppressive shadows of our current ignorance. I still tell Heisenberg’s story. It never fails to speak to me from this deep place of reassurance.

I believe that we have only just begun the process of discovering and inventing the new organizational forms that will inhabit the twenty-first century. To be responsible inventors and discoverers, we need the courage to let go of the old world, to relinquish most of what we have cherished, to abandon our interpretations about what does and doesn’t work. We must learn to see the world anew. As Einstein is often quoted as saying: No problem can be solved from the same consciousness that created it.

There are many places to search for new answers in a time of paradigm shifts. For me, it was appropriate that my inquiry led back to the natural sciences, reconnecting me to an earlier vision of myself. At fourteen, I aspired to be a space biologist and carried heavy astronomy texts on the New York subway to weekly classes at the Hayden Planetarium. These texts were far too dense for me to understand, but I carried them anyway because they looked so impressive. My abilities in biology were better founded, and I began college majoring in biology, but my encounters with advanced chemistry ended that career, and I turned to the greater ambiguity of the social sciences. Like many social scientists, I am at heart a lapsed scientist, still hoping the world will yield up its secrets to me.

But my focus on science is more than a personal interest. Each of us lives and works in organizations designed from Newtonian images of the universe. We manage by separating things into parts, we believe that influence occurs as a direct result of force exerted from one person to another, we engage in complex planning for a world that we keep expecting to be predictable, and we search continually for better methods of objectively measuring and perceiving the world. These assumptions, as I explain in Chapter Two, come to us from seventeenth-century physics, from Newtonian mechanics. They are the basis from which we design and manage organizations, and from which we do research in all of the social sciences. Intentionally or not, we work from a world view that is strongly anchored in the natural sciences.

But the science has changed. If we are to continue to draw from science to create and manage organizations, to design research, and to formulate ideas about organizational design, planning, economics, human motivation, and change processes (the list can be much longer), then we need to at least ground our work in the science of our times. We need to stop seeking after the universe of the seventeenth century and begin to explore what has become known to us during the twentieth century. We need to expand our search for the principles of organization to include what is presently known about how the universe organizes.

The search for the lessons of new science is still in progress, really in its infancy; but what I hope to convey in these pages is the pleasure of sensing those first glimmers of a new way of thinking about the world and its organizations. The light may be dim, but its potency grows as the door cracks wider and wider. Here there are scientists who write about natural phenomena with a poetry and a clarity that speak to dilemmas we find in organizations. Here there are new images and metaphors for thinking about our own organizational experiences. This is a world of wonder and not knowing, where many scientists are as awestruck by what they see as were the early explorers who marveled at new continents. In this realm, there is a new kind of freedom, where it is more rewarding to explore than to reach conclusions, more satisfying to wonder than to know, and more exciting to search than to stay put. Curiosity, not certainty, becomes the saving grace.

This is not a book filled with conclusions, cases, or exemplary practices. It is deliberately not that kind of book, for two reasons. First, I don’t believe that organizations are ever changed by imposing a model developed elsewhere. So little transfers to, or inspires, those trying to work at change in their own organizations. In every organization, we need to look internally, to see one another as the critical resources on this voyage of discovery. We need to learn how to engage the creativity that exists everywhere in our organizations. Second, the new physics cogently explains that there is no objective reality out there waiting to reveal its secrets. There are no recipes or formulas, no checklists or expert advice that describe “reality.” If context is as crucial as the science explains, then nothing really transfers; everything is always new and different and unique to each of us. We must engage with each other, experiment to find what works for us, and support one another as the true inventors that we are.

This book attempts to be true to that new vision of reality, where ideas and information are but half of what is required to evoke reality. The creative possibilities of the ideas represented here depend on your engagement with them. I assigned myself the task of presenting material to provoke and engage you, knowing that your experience with these pages will produce different ideas, different hopes, and different experiments than mine. It is not important that we agree on one expert interpretation or one best practice. That is not the nature of the universe in which we live. We inhabit a world that co-evolves as we interact with it. This world is impossible to pin down, constantly changing, and infinitely more interesting than anything we ever imagined.

Though the outcomes to be gained from reading this book are unique to each of you, the ideas I have chosen to think about focus on the meta-issues that concern those of us who work in organizations: Where is order to be found? How do complex systems change? How do we create structures that are flexible and adaptive, that enable rather than constrain? How do we simplify things without losing what we value about complexity? How do we resolve personal needs for autonomy and growth with organizational needs for prediction and accountability?

The new science research referred to in this book comes from the disciplines of physics, biology, and chemistry, and from theories of evolution and chaos that span several disciplines. Each chapter inquires into metaphorical links between certain scientific perspectives and organizational phenomena, but it may be useful first to say something about the direction of new science.

Scientists in many different disciplines are questioning whether we can adequately explain how the world works by using the machine imagery emphasized in the seventeenth century by such great geniuses as Sir Isaac Newton and René Descartes. This machine imagery leads to the belief that studying the parts is the key to understanding the whole. Things are taken apart, dissected literally or figuratively (as we have done with business functions, academic disciplines, areas of specialization, human body parts), and then put back together without any significant loss. The assumption is that the more we know about the workings of each piece, the more we will learn about the whole.

Newtonian science ...