Order for Free March 1998

Life -- the Artificial and the Real

by Jim Highsmith

On the desk in my office sits EEK, an entirely self-contained, enclosed, living ecosystem. Five inches high, EEK contains algae, very small shrimp and snails, and multitudes of microscopic bacteria, all living by exchanging stuff with each other or converting light to biochemical energy. Meanwhile, on the monitor beside EEK, a forest of digital beings (Biots) exchange digital stuff. They live, eat (each other of course), mate, give birth, evolve into new organisms, and die—a sea of artificial life created in silicon by one of the available artificial life programs.

I use these two visions of life, the real and the artificial, as constant reminders of a new way of thinking about complex systems, a new way of thinking about how complex software products come into being. In the last ten years, software has entered a new frontier, a place where the frequency of change and the speed of the market have created a raw, hostile environment for software development, one where the basic tenants of process improvement, software engineering, and command & control management are insufficient for success. It is a frontier, I believe, where we can learn something by studying life itself and its mechanisms for adaptation and evolution.

Within the last decade, the study of complexity, or complex adaptive systems, in fields such as evolutionary biology, ecology, and physics has begun to spill over into the management of organizations. Some of these concepts help explain why leading edge, high-tech companies have succeeded by breaking historic patterns of behavior, and gives us some insight into how organizations may operate more effectively in the future.

This article introduces some fundamental concepts about how adaptation, through spontaneous self-organization, may help us work in this world of extreme high-change and high-speed. It then explores how organizations, in this era of the Internet and massive connectivity, might take advantage of these concepts. In essence, I want to explore two questions: What is spontaneous, emergent order? And, how to we get it?

Adaptation, not Optimization

In a slower paced, less complex world, the remnants of the Newtonian age work to perfection. Optimization, through rigorous process improvement, detailed task control based on predicted outcomes, and hierarchical management have been enormously successful. In our industry, the pinnacle of optimization may be the Software Engineering Institute's Capability Maturity Model Level 5 organizations—optimized to produce the known at a high level of efficiency. But if CMM Level 5 is the pinnacle, why are there so many very successful software companies who can’t even spell CMM? Are they just lucky, or is something else at work?

What about the unknown? What if we don't know exactly what product the market wants? What if competitor's products disrupt our carefully laid plans? What if a new technology obsoletes our product development efforts? What if we have some general sense of direction, but there is a lot of fogginess in that direction? What if we are facing an impossible delivery schedule? Optimization has limited application in these situations. Companies and software development groups in fast moving markets need a different approach. In these environment, "Adaptation is significantly more important than optimization [1]." Furthermore, in this context, adaptation is far more than another word for change management.

What is Self-Organization (Order for Free)?

As software engineers, we tend to approach the problem of building complex software products, from the perspective of engineers. By in large, engineering is based on predictable events, deterministic patterns, and linear construction—overall by an attitude of, "I can make it happen." We believe larger more complex problems need a rigorous, engineering discipline— an imposed order.

What if there is another source of order in the world? What if there is both imposed order and a second, spontaneously arising, emergent order. Biologist Stuart Kauffman has spent more than thirty years trying to answer this question for scientific areas more complex than even Windows NT. Kauffman uses Genetic (Boolean) network simulations to study the complexities of biological evolution.

Kauffman’s basic premise is that natural selection is necessary, but not sufficient, to have evolved complex organisms. Natural selection, by itself, faces awesome mathematical odds. In a 100,000 node genetic simulation (the approximate number of human genes), the potential set of states is hypervast—1030,000. However, in certain situations, Kauffman found the model "would settle down and cycle through a tiny, tiny state cycle with a mere 317 states on it. Order for free. … Natural selection is important, but another source—self-organization—is the root source of order [2]."

Self-organization is a property of complex adaptive systems similar to a collective "aha," that moment of creative energy when the solution to some nagging problem emerges. Self-organization arises when individual, independent agents (cells in a body, species in an ecosystem, developers in a feature team) cooperate to create emergent outcomes. An emergent outcome is a property beyond the capability of any individual agent. For example, individual neurons in the brain do not possess consciousness, but collectively the property of consciousness emerges. We tend to view this phenomena of collective emergence as accidental, or at least unruly and undependable. The study of self-organization is proving that view to be wrong.

Kauffman’s investigations into biological complexity offer insight into managing organizational complexity. There are those who think the insights are far reaching. Kevin Kelly, the executive editor of Wired magazine, devoted an entire chapter to Kauffman’s work in his own fascinating book, Out of Control: The New Biology of Machines, Social Systems, and the Economic World, in part about the future of organizations.

How Do We Get It?

As natural selection is necessary, but insufficient, to create biological complexity, similarly optimization is necessary but insufficient to create products in complex environments.

Achieving emergent results, in an organizational context, requires:

An adaptive mindset—accepting the limits of imposed order,
Revising development practices to support self-organization, ie. an Adaptive Lifecycle,
Re-thinking management—especially how we create collaborative environments.

There has been a growing emphasis on collaboration, but just maybe, collaboration is really more important than we realize. Can it really be as much a source of order in our complex interactions as discipline and engineering? If so, is are the resources we are spending on our collaboration networks nearly enough?

Many software engineers feel un-alone time is mostly wasted. Collaboration, to their way of thinking, may be characterized by one or more of the following:

Overpowering others in meetings with their own towering intellect,
Wasting time in meetings when they should be back in their cube coding, testing, etc.,
Attaching an alias to every email,
For others who don’t have a clue about what is going on,
Touchy-feely, time-wasting meetings cooked up by the bozos in Human Resources,
Creating a project Intranet page.

Michael Schrage offers my favorite definition of collaboration, "The act of collaboration is an act of shared creation and/or discovery [3]." Communication is passive—I send you something. Collaboration is active—we jointly create something. At its best, collaboration produces more than the sum of the parts. For other than trivial products, collaboration is not an option. How well we collaborate is the variable.

This raises a variety of very interesting questions about collaboration. The initial questions are important, first, how can we improve the collaborative efforts of single, co-located feature teams? Second, how can we improve the collaborative efforts of multiple, virtual feature teams? Answers to these questions involve both interpersonal and informational dimensions.

But complex systems research is pointing us towards even more fundamental issues. Is an organization’s ability to adapt quickly a direct function of its ability to collaborate? And, more pointedly, is that ability to adapt a function of an organization’s collaborative structure?

The process approach (change management, change agents, changing an organizations culture) has gained immense popularity in recent years as companies try to keep up with rapid change. Small companies in rapidly changing markets seem to change fairly rapidly, without much ado about formal change processes. The more interesting phenomena is larger organizations, such as Microsoft, who operate in rugged market landscapes and exhibit adaptability without extensive formal change process efforts. Why? A first response may be the market demands nimbleness to survive. But that does not explain why certain companies are better at adapting.

One implication to be drawn, from both theory and experience, is:

the structure of an organization’s collaboration network has significant impact on its ability to produce emergent results and ultimately on its very ability to adapt.

Companies establish skunk works to foster innovation because the structure of the larger organization is too rigid. Netscape, however, doesn’t have a skunk works, it is one. Could it be that much of the effort applied to change processes in larger organizations is wasted because their collaboration structure is not conducive to adaptation? Should these firms be spending at least as much time experimenting with their collaboration network structure as the latest change management technique? Many firms are spending millions trying to promote change, when their collaboration structure is solidly locked into stability.

We have had hundreds of years to perfect hierarchical management structures, but less than twenty to deal with networked organizations. The movement in many of today’s companies towards a flattened hierarchy to deal with administrative issues, and networked teams to actually produce products is being done with little data about how to tune those networks for effectiveness. This is one area where a better understanding of complex adaptive systems may provide significant payback.

Tuning Collaborative Networks

Complexity research, and my own experiences with software development teams, provide several ideas for tuning collaboration networks.

First, it appears that above a certain point, the number of connections decreases adaptation. While the absolute number of connections is the key parameter in Boolean networks, organizational connections would include characteristics such as flow, diversity, and richness of the connection. Complexity studies also indicate adaptation occurs at the edge of chaos, a transition zone between confusion and stagnation. Kauffman’s work indicates not only is this edge of chaos real, but it appears to truly be a fairly sharp edge. The optimal number of connections for balance appears to be relatively small.

This relatively low number of connections per node does not seem to vary much with network size. As networks get larger, as more nodes are added, the connections to each node need to remain relatively constant. In today’s networked companies where a "hook um up, link um up, ride um out" mentality prevails, we may be unwittingly pushing ourselves into the chaotic zone, where emergent results are lost in the noise.

Kauffman’s work also analyzes the complexity of the rule base within a node. If the goal is to balance at the creative edge, it appears overly complex rules override the benefits conveyed by connectivity. Although we know overly rigorous rules stifle creativity, we need a better understanding of how rules and connections interact to keep organizations poised at the edge.

Finally, there is a clear linkage between organisms and the ruggedness of the competitive landscapes in which they adapt. As markets stabilize and companies settle into the ordered realm, they move away from the edge and become resistant to change. When the market changes suddenly, they cannot catch up—their control hierarchies have ousted their collaborative networks in the power struggle. Bringing in the local culture change guru will not be enough to overcome their structural deficiencies.

Finale

As I sit, contemplating how to finish this article, one of the shrimp in my ecosystem swims madly around in circles. I wonder if the link between biology, complex adaptive systems, organizations, and collaborative networks is caught up in the same kind of meaningless swirl. But I am heartened by Kauffman, "There are times in any science when one senses that a transformation to deeper understanding is pressing upward in some as yet poorly articulated form. We may be in such a period in biology." I submit we may be in the same position with respect to organizations in the networked era. The linkage between complex adaptive systems concepts and organizational performance still lies in the area of poorly articulated form. But our needs will not always wait for the well articulated and the proven—it is part of surfing on the leading wave.

Adaptation does not replace optimization, they are complementary. It is not either, or, but a balance, a teetering on the edge to gain the best of both. However, most organizations understand optimization, but confuse adaptation with adhocracy. As complexity increases, the failure of optimization is met with a fervent call for greater optimization, for more imposed order—exactly the wrong strategy. A better understanding of adaptation as a key to combating complexity, and a new perspective on collaboration as an activating mechanism, should help restore a balance—to both produce order for free and create more agile companies.

=============================================================================

1 Jim Highsmith, Adaptive Software Development: An Evolutionary Approach to Managing Complex Systems, Dorset House Publishing, 1998.

2 Stuart Kauffman, At Home in the Universe: The Search for the Laws of Self-Organization and Complexity, Oxford University Press, 1995.

3 Michael Schrage, No More Teams: Mastering the Dynamics of Creative Collaboration, Currency Doubleday, 1989.