In a famous episode in the “I Love Lucy” television series—”Job Switching,” better known as the chocolate factory episode—Lucy and her best-friend coworker Ethel are tasked to wrap chocolates flowing by on a conveyor belt in front of them. Each time they get better at the task, the conveyor belt speeds up. Eventually they cannot keep up and the whole scene collapses into chaos.
The threshold between order and chaos seems thin. A small perturbation—such as a slight increase in the speed of Lucy’s conveyor belt—can either do nothing or it can trigger an avalanche of disorder. The speed of events within an avalanche overwhelms us, sweeps away structures that preserve order, and robs our ability to function. Quite a number of disasters, natural or human-made, have an avalanche character—earthquakes, snow cascades, infrastructure collapse during a hurricane, or building collapse in a terror attack. Disaster-recovery planners would dearly love to predict the onset of these events so that people can safely flee and first responders can restore order with recovery resources standing in reserve.
Disruptive innovation is also a form of avalanche. Businesses hope their new products will “go viral” and sweep away competitors. Competitors want to anticipate market avalanches and side-step them. Leaders and planners would love to predict when an avalanche might occur and how extensive it might be.
In recent years complexity theory has given us a mathematics to deal with systems where avalanches are possible. Can this theory make the needed predictions where classical statistics cannot? Sadly, complexity theory cannot do this. The theory is very good at explaining avalanches after they have happened, but generally useless for predicting when they will occur.
Complexity Theory
In 1984, a group of scientists founded the Santa Fe Institute to see if they could apply their knowledge of physics and mathematics to give a theory of chaotic behavior that would enable professionals and managers to move productively amid uncertainty. Over the years the best mathematical minds developed a beautiful, rich theory of complex systems.
Traditional probability theory provides mathematical tools for dealing with uncertainty. It assumes the uncertainty arises from random variables that have probability distributions over their possible values. It typically predicts the future values of the variable by computing a mean of the distribution and a confidence interval based on its standard deviation. For example, in 1962 Everett Rogers studied the adoption times of the members of a community in response to a proposed innovation.5 He found they follow a Normal (Bell) curve that has a mean and a standard deviation. A prediction of adoption time is the mean time bracketed by a confidence interval: for example, 68% of the adoption times are within one standard deviation of the mean and 95% are within two standard deviations.
In 1987, researchers Per Bak, Chao Tang, and Kurt Wiesenfeld published the results of a simple experiment that demonstrated the essence of complexity theory.4 They observed a sand pile as it formed by dropping grains of sand on a flat surface. Most of the time, each new grain would settle into a stable position on the growing cone of sand. But at unpredictable moments a grain would set off an avalanche of unpredictable size that cascaded down the side of the sand pile. The researchers measured the time intervals between avalanche starts and the sizes of avalanches. To their surprise, these two random variables did not fit any classical probability distribution such as the Normal or Poisson distributions. Instead, their distributions followed a “power law,” meaning the probability of a sample of length x is proportional to x−k, where k a fixed parameter of the random process. Power law distributions have a finite mean only if k>2 and variance only if k>3. This means a power law with k≤2 has no mean or variance. Its future is unpredictable. When 2<k≤3, the mean is finite but not the confidence interval. Bak et al. had discovered something different—a random process whose future could not be predicted with any confidence.
This was not an isolated finding. Most of the random processes tied to chaotic situations obey a power law with k≤3. For example, the appearance of new connections among Web pages is chaotic. The number of Web pages with x connections to other pages is proportional to 1/x2—the random process of accumulating links produces 1/4 as many pages with 2x connections as with x connections. This was taken as both bad and good news for the Internet. The bad news is that because there are a very few “hubs”—servers hosting a very large number of connections—an attacker could shatter the network into isolated pieces by bringing down the hubs. The good news is the vast majority of servers host few connections and thus random server failures are unlikely to shatter the network. What makes this happen is “preferential attachment”—when a new Web page joins the network, it tends to connect with the most highly connected nodes already in the network. Startup company founders try to plot strategies to bring about rapid adoption of their technologies and transform their new services into hubs.
Hundreds of processes in science and engineering follow power laws and their key variables are unpredictable. Innovation experts believe innovations follow a power law—the number of innovations adopted by communities of size x is proportional to x−2—not good news for startup companies hoping to predict their innovations will take over the market.
Hundreds of processes in science and engineering follow power laws and their key variables are unpredictable.
Later Bak1 developed a theory of unpredictability that has subsequently been copied by popular writers like Nassim Nicholas Taleb and others.6 Bak called it punctuated equilibrium, a concept first proposed by Stephen Jay Gould and Niles Eldredge in 1972.3 The idea is that new members can join a complex system by fitting in to the existing structure; but occasionally, the structure passes a critical point and collapses and the process starts over. The community order that has worked for a long time can become brittle. Avalanche is an apt term for the moment of collapse. In the sand pile, for example, most new grains lodge firmly into a place on the pile but occasionally one sets off an avalanche that changes the structure. In the Internet, malware can quickly travel via a hub to many nodes and cause a large-scale avalanche of disruption. In an economy, a new technology can suddenly trigger an avalanche that sweeps away an old structure of jobs and professions and establishes a new order, leaving many people stranded. Complexity theory tells us we frequently encounter systems that transition between stability and randomness.
Figure. Log-log plot of the exceedance versus intervals between terror attacks follows a straight line. Exceedance is the probability that an interval is greater than x (a tail of the distribution). A straight line on log-log plot is the signature of a power law; here the slope is −1.4, telling us the tails of the distribution are a power law y=x−1.4. Because 1.4 is less than 2, this distribution has no finite mean or standard deviation: the time to next terror attack is unpredictable.
Punctuated equilibrium appears differently in different systems because self-organization manifests in different ways. In the Internet, it may be the vulnerability to the failure of highly connected hubs. In a national highway system, it may be the collapse of maintenance as more roads are added, bringing new traffic that deteriorates old roads faster. In geology, it may be the sudden earthquake that shatters a stable fault and produces a cascade of aftershocks. In a social system, it may be the outbreak of protests when people get “fed up.”
Explanations but Not Predictions
What can we learn from all this? Many systems have a strong social component, which leads to forms of preferential attachment and power-laws governing the degrees of connectivity in the social network. These systems are susceptible to sudden changes of structure of unpredictable onset and extent. The best we can say is the conditions for avalanche are present but we cannot say with any certainty the avalanche will actually happen or, if it does, what its extent will be. In other words, we are able to explain an avalanche after it happens but we are profoundly unable to predict anything about it before it happens.
Earthquake preparedness is an example in nature that does not depend on humans. Seismic experts can tell us where the fault lines are and compute the probabilities of earthquake on different faults. They cannot, however, predict when an earthquake will happen or how large it will be. In effect they are trying to predict when an earthy avalanche—collapse of structure in a section of earth’s crust—will happen. Similarly, snow experts know when conditions are “ripe” for an avalanche and can call for evacuating the area. But they cannot know exactly where a snow avalanche may start, or when, or how much snow will sweep down. These experts call on people to be prepared but few actually heed the advice and lay in necessary supplies or make necessary contingency plans.
Navigating in Uncertainty
Complexity researchers have turned to simulations of complex systems to see when avalanches happen and how large they are. These simulations often reveal regularities in the state spaces of those systems that can be usefully exploited to make predictions.
One of the most difficult environments to navigate is the social space in which we perform our work. This space is dominated by choices that other people make beyond our control.
What are more pragmatic things we can do to cope with uncertainty? We can learn some lessons from those who must deal with disasters such as fires, earthquakes, floods, or terror attacks. Their data shows the times between events and sizes of events follow power laws and cannot be predicted. Their coping strategy boils down to preparedness and resiliency. Preparedness means to have recovery resources standing by in case of need. Resiliency means to rapidly bounce back and restore order and function.
They have worked out strategies to identify the situations most “ripe” for an avalanche. For instance, the power law for terror attacks shows that attacks tend to cluster in time at a given location Thus, a next attack is more likely at the same location as the current attack The preparedness strategies include rapid mobilization of law enforcement just after an attack to counter the tendency for a new attack, and to identify optimal geographic locations for positioning recovery resources and supplies. Resilience strategies include rapidly mobilizing technicians and artisans to restore broken communications and facilities.
Uncertainty in Professional Work
What can we do when we find ourselves in chaotic situations and must still navigate through the uncertainty to achieve our goals?
One of the most difficult environments to navigate is the social space in which we perform our work. This space is dominated by choices that other people make beyond our control. When we propose innovations, we are likely to encounter resistance from some sectors of our community that do not want the innovation; they can be quite inventive in finding ways to block our proposals.2 When we start new projects or even companies, we do not know whether our plans are going to take off or just wither away. Even in normal everyday working environments, conflicts and contingencies suddenly arise and we must resolve them to keep moving forward.
The analogy of a surfer is useful in approaching these situations. A surfer aims to ride the waves to the shore without losing balance and being swept under. The waves can be turbulent and unpredictable. The surfer must maintain balance, ride the crests moving toward the shore, and dodge side waves and cross currents. The surfer may need to jump to a new wave when the time is right, or quickly tack to avoid an unfavorable current or wind. Thus, the surfer generates a path through the turbulent waves.
In the social space, waves manifest as groups of people disposed to move in certain directions and not in others—sometimes the waves appear as fads or “memes” and they have a momentum that is difficult to divert. As a professional, we become aware of these waves and try to harness them to carry us toward our goal. As each surprise pops up, we instinctively look for openings into which we can move—and, more importantly, we create openings by starting conversations that assuage the concerns of those whose resistance threatens to block us. These little deals cut a path through the potential resistance and get us to our goal.
The lesson here is that we listen for the waves, ride their momentum toward our goal, and make adjustments by creating openings in our conversations with other people. At its best, the complexity theory helps us understand when a process is susceptible to unpredictable avalanches. We move beyond the limitations of the theory by generating openings in our conversations with other people.
Join the Discussion (0)
Become a Member or Sign In to Post a Comment