|Order for Free||March 1998|
Life -- the Artificial and the Real
by Jim Highsmith
On the desk in my office sits EEK, an entirely self-contained, enclosed, living ecosystem. Five inches high, EEK contains algae, very small shrimp and snails, and multitudes of microscopic bacteria, all living by exchanging stuff with each other or converting light to biochemical energy. Meanwhile, on the monitor beside EEK, a forest of digital beings (Biots) exchange digital stuff. They live, eat (each other of course), mate, give birth, evolve into new organisms, and diea sea of artificial life created in silicon by one of the available artificial life programs.
I use these two visions of life, the real and the artificial, as constant reminders of a new way of thinking about complex systems, a new way of thinking about how complex software products come into being. In the last ten years, software has entered a new frontier, a place where the frequency of change and the speed of the market have created a raw, hostile environment for software development, one where the basic tenants of process improvement, software engineering, and command & control management are insufficient for success. It is a frontier, I believe, where we can learn something by studying life itself and its mechanisms for adaptation and evolution.
Within the last decade, the study of complexity, or complex adaptive systems, in fields such as evolutionary biology, ecology, and physics has begun to spill over into the management of organizations. Some of these concepts help explain why leading edge, high-tech companies have succeeded by breaking historic patterns of behavior, and gives us some insight into how organizations may operate more effectively in the future.
This article introduces some fundamental concepts about how adaptation, through spontaneous self-organization, may help us work in this world of extreme high-change and high-speed. It then explores how organizations, in this era of the Internet and massive connectivity, might take advantage of these concepts. In essence, I want to explore two questions: What is spontaneous, emergent order? And, how to we get it?
In a slower paced, less complex world, the remnants of the Newtonian age work to perfection. Optimization, through rigorous process improvement, detailed task control based on predicted outcomes, and hierarchical management have been enormously successful. In our industry, the pinnacle of optimization may be the Software Engineering Institute's Capability Maturity Model Level 5 organizationsoptimized to produce the known at a high level of efficiency. But if CMM Level 5 is the pinnacle, why are there so many very successful software companies who cant even spell CMM? Are they just lucky, or is something else at work?
What about the unknown? What if we don't know exactly what product the market wants? What if competitor's products disrupt our carefully laid plans? What if a new technology obsoletes our product development efforts? What if we have some general sense of direction, but there is a lot of fogginess in that direction? What if we are facing an impossible delivery schedule? Optimization has limited application in these situations. Companies and software development groups in fast moving markets need a different approach. In these environment, "Adaptation is significantly more important than optimization ." Furthermore, in this context, adaptation is far more than another word for change management.
As software engineers, we tend to approach the problem of building complex software products, from the perspective of engineers. By in large, engineering is based on predictable events, deterministic patterns, and linear constructionoverall by an attitude of, "I can make it happen." We believe larger more complex problems need a rigorous, engineering discipline an imposed order.
What if there is another source of order in the world? What if there is both imposed order and a second, spontaneously arising, emergent order. Biologist Stuart Kauffman has spent more than thirty years trying to answer this question for scientific areas more complex than even Windows NT. Kauffman uses Genetic (Boolean) network simulations to study the complexities of biological evolution.
Kauffmans basic premise is that natural selection is necessary, but not sufficient, to have evolved complex organisms. Natural selection, by itself, faces awesome mathematical odds. In a 100,000 node genetic simulation (the approximate number of human genes), the potential set of states is hypervast1030,000. However, in certain situations, Kauffman found the model "would settle down and cycle through a tiny, tiny state cycle with a mere 317 states on it. Order for free. Natural selection is important, but another sourceself-organizationis the root source of order ."
Self-organization is a property of complex adaptive systems similar to a collective "aha," that moment of creative energy when the solution to some nagging problem emerges. Self-organization arises when individual, independent agents (cells in a body, species in an ecosystem, developers in a feature team) cooperate to create emergent outcomes. An emergent outcome is a property beyond the capability of any individual agent. For example, individual neurons in the brain do not possess consciousness, but collectively the property of consciousness emerges. We tend to view this phenomena of collective emergence as accidental, or at least unruly and undependable. The study of self-organization is proving that view to be wrong.
Kauffmans investigations into biological complexity offer insight into managing organizational complexity. There are those who think the insights are far reaching. Kevin Kelly, the executive editor of Wired magazine, devoted an entire chapter to Kauffmans work in his own fascinating book, Out of Control: The New Biology of Machines, Social Systems, and the Economic World, in part about the future of organizations.
As natural selection is necessary, but insufficient, to create biological complexity, similarly optimization is necessary but insufficient to create products in complex environments.
Achieving emergent results, in an organizational context, requires:
|An adaptive mindsetaccepting the limits of imposed order,|
|Revising development practices to support self-organization, ie. an Adaptive Lifecycle,|
|Re-thinking managementespecially how we create collaborative environments.|
|Overpowering others in meetings with their own towering intellect,|
|Wasting time in meetings when they should be back in their cube coding, testing, etc.,|
|Attaching an alias to every email,|
|For others who dont have a clue about what is going on,|
|Touchy-feely, time-wasting meetings cooked up by the bozos in Human Resources,|
|Creating a project Intranet page.|
Companies establish skunk works to foster innovation because the structure of the larger organization is too rigid. Netscape, however, doesnt have a skunk works, it is one. Could it be that much of the effort applied to change processes in larger organizations is wasted because their collaboration structure is not conducive to adaptation? Should these firms be spending at least as much time experimenting with their collaboration network structure as the latest change management technique? Many firms are spending millions trying to promote change, when their collaboration structure is solidly locked into stability.
We have had hundreds of years to perfect hierarchical management structures, but less than twenty to deal with networked organizations. The movement in many of todays companies towards a flattened hierarchy to deal with administrative issues, and networked teams to actually produce products is being done with little data about how to tune those networks for effectiveness. This is one area where a better understanding of complex adaptive systems may provide significant payback.
Complexity research, and my own experiences with software development teams, provide several ideas for tuning collaboration networks.
First, it appears that above a certain point, the number of connections decreases adaptation. While the absolute number of connections is the key parameter in Boolean networks, organizational connections would include characteristics such as flow, diversity, and richness of the connection. Complexity studies also indicate adaptation occurs at the edge of chaos, a transition zone between confusion and stagnation. Kauffmans work indicates not only is this edge of chaos real, but it appears to truly be a fairly sharp edge. The optimal number of connections for balance appears to be relatively small.
This relatively low number of connections per node does not seem to vary much with network size. As networks get larger, as more nodes are added, the connections to each node need to remain relatively constant. In todays networked companies where a "hook um up, link um up, ride um out" mentality prevails, we may be unwittingly pushing ourselves into the chaotic zone, where emergent results are lost in the noise.
Kauffmans work also analyzes the complexity of the rule base within a node. If the goal is to balance at the creative edge, it appears overly complex rules override the benefits conveyed by connectivity. Although we know overly rigorous rules stifle creativity, we need a better understanding of how rules and connections interact to keep organizations poised at the edge.
Finally, there is a clear linkage between organisms and the ruggedness of the competitive landscapes in which they adapt. As markets stabilize and companies settle into the ordered realm, they move away from the edge and become resistant to change. When the market changes suddenly, they cannot catch uptheir control hierarchies have ousted their collaborative networks in the power struggle. Bringing in the local culture change guru will not be enough to overcome their structural deficiencies.
As I sit, contemplating how to finish this article, one of the shrimp in my ecosystem swims madly around in circles. I wonder if the link between biology, complex adaptive systems, organizations, and collaborative networks is caught up in the same kind of meaningless swirl. But I am heartened by Kauffman, "There are times in any science when one senses that a transformation to deeper understanding is pressing upward in some as yet poorly articulated form. We may be in such a period in biology." I submit we may be in the same position with respect to organizations in the networked era. The linkage between complex adaptive systems concepts and organizational performance still lies in the area of poorly articulated form. But our needs will not always wait for the well articulated and the provenit is part of surfing on the leading wave.
Adaptation does not replace optimization, they are complementary. It is not either, or, but a balance, a teetering on the edge to gain the best of both. However, most organizations understand optimization, but confuse adaptation with adhocracy. As complexity increases, the failure of optimization is met with a fervent call for greater optimization, for more imposed orderexactly the wrong strategy. A better understanding of adaptation as a key to combating complexity, and a new perspective on collaboration as an activating mechanism, should help restore a balanceto both produce order for free and create more agile companies.
1Jim Highsmith, Adaptive Software Development: An Evolutionary Approach to Managing Complex Systems, Dorset House Publishing, 1998.
2Stuart Kauffman, At Home in the Universe: The Search for the Laws of Self-Organization and Complexity, Oxford University Press, 1995.
3Michael Schrage, No More Teams: Mastering the Dynamics of Creative Collaboration, Currency Doubleday, 1989.