Table of Contents This Issue | 1999 Issues | Other Years
© Copyright 1999 Cutter Information Corp. All rights reserved. No part of this document may be
reproduced in any manner without express written permission from Cutter Information Corp.
ADAPTIVE MANAGEMENT: PATTERNS FOR THE E-BUSINESS ERA
by Jim Highsmith
In the mid-1970s, Bob Taylor, Alan Kay, and others at Xerox PARC articulated a vision of distributed personal computing. The PC revolution of the 1980s achieved the "personal" aspect of the vision, while in the 1990s, the Internet phenomenon is completing the "distributed" side. IT managers, swept up in the sheer magnitude and difficulty of the business strategy and technology transformation resulting from this distributed-computing idea, haven't fully considered its impact on their own organizations. The same software technology that has enabled decreasing time to market, catapulted us into the era of e- business, and created intense competitive turbulence also impacts the very essence of what IT organizations do and how they are managed. We, the technologists, extol to executives how the Internet and e-business are going to change the very core of their business, and at the same time we blithely assume the status quo within IT management. The future is not going to work out that way.1
Most IT organizations are steeped in a tradition of optimization, efficiency, predictability, control, rigor, and process improvement. The emerging e-business economy requires adaptability, speed, collaboration, improvisation, flexibility, innovation, and suppleness. For example, outsourcing (at least that component of outsourcing driven by business-executive frustration with IT) is not a cure, but a symptom -- one of the symptoms of trying to survive in the 21st century with a 1970s management model. In the next 10 years, the gut-wrenching changes in IT organizations will not be in implementing such highly hyped technologies as component development or enterprise application integration, but in overhauling the pivotal leadership practices required to grow adaptable organizations.
Observe the management style in mainline IT organizations versus that in Microsoft, Sun, Netscape, or Amazon.com. These high-tech companies didn't pick their management style because it felt good; market turbulence drove them there. "Successful firms in fiercely competitive and unpredictably shifting industries pursue a competing on the edge strategy," write Shona Brown and Kathleen Eisenhardt in Competing on the Edge: Strategy as Structured Chaos.2 "The goal of this strategy is not efficiency or optimality in the usual sense. Rather, the goal is flexibility -- that is, adaptation to current change and evolution over time, resilience in the face of setbacks, and the ability to locate the constantly changing sources of advantage. Ultimately, it means engaging in continual reinvention." We need to focus less on what organizations "ought" to do and focus more on what is actually working in the rough and tumble real world.
Proponents of optimization, from business process reengineering (BPR) to the SEI's CMM, denigrate anything that does not fit their model as "immature" and "undisciplined." They consider concepts such as "good-enough software" to be vile. But let's look at results, starting with the fact that BPR has been an unmitigated failure. Furthermore, if we view each CMM level as a "product" and acknowledge that fewer than 10 luxury models (i.e., Level 5) have been sold in 15 years, then how can we view the CMM as successful? Maybe the reason the CMM isn't more widely used is not our immaturity, but the fact that our world is not stable enough for optimizing practices based on assumptions of controllability and predictability to deliver the results we need.
On the flip side, software applications -- from Web sites to ERP systems to e-business and e-commerce -- are phenomenally successful. Yet we berate our own success with scare tactics like the annual Standish Group's "Chaos" report, which always sounds as though a disaster has struck. While we can always do better, maybe we're much more successful than we give ourselves credit for. Maybe it's Standish's metrics that are faulty. Of course we need the negative metrics to justify the money spent on process improvement programs.
Rather than dwell on the negative press about software development, we need to focus on a new management model to survive the future's turbulence. The model needs to incorporate the best practices of good software engineering and project management, but within an adaptive rather than an optimizing framework. Rather than processes, the model needs to focus on "patterns." We need to move from a 20th century "Command-Control" model to a 21st century "Leadership- Collaboration" one.
What we name things is important. Traditional management has been characterized by the term "Command-Control." It is steeped in the traditions of the military, Frederick Taylor's scientific management, and William Whyte's "Organization Man" of the 1950s. While new terms have spewed forth from the pages of the Harvard Business Review and management books -- empowerment, participative management, learning organization, human-centered management, and the like -- none has encompassed the breadth, or depth, I think necessary for managing modern organizations. Dee Hock, former CEO of VISA, made a stab with his "chaordic" management (balanced between chaos and order), but the word doesn't roll off the tongue easily.
So I use the phrase "Leadership-Collaboration" (L-C) to name this new style of management. Today, we need leaders more than commanders. Leaders, by my definition at least, understand that they occasionally need to command, but that's not their predominant style. Leaders provide direction and create environments where talented people can be innovative, creative, and make effective decisions.
Commanders know the objective; leaders grasp the direction. Commanders dictate; leaders influence. Controllers demand; collaborators facilitate. Controllers micro-manage; collaborators encourage. Managers who embrace the L-C model understand that their primary role is to set direction, to provide guidance, and to facilitate in connecting people and teams. The L-C model encompasses the basic philosophical assertion that in turbulent environments "adaptation" is more important than "optimization." However, becoming adaptive is not easy. The shift from optimization to adaptation -- for an organization or a project -- requires a profound cultural shift. Redefining existing terminology isn't enough; we have to use new words that embrace new ideas. The shift from "process" to "pattern" is one of these.
FROM PROCESSES TO PATTERNS
Let's throw away the word process and substitute pattern. Process is a mechanical word, derived from a view that the world is predictable and controllable. "Process" carries too much baggage. BPR efforts failed for a variety of reasons, but primarily because proponents forgot about people. Substitute people for process -- i.e., business people reengineering -- and we suddenly uncover the depth of our misconceptions. Business "people" resist being reengineered. Software people (I guess the term would be "software people reengineering") resist also. People and process are not synonyms.
Our use of the term "process" is derived from a cybernetic input-process- output-control view of organizations in which we measure the output, compare it to plans, and tweak our control levers to maintain a carefully measured and repeatable process. Visit the first page of the Software Engineering Institute's Web site, and the terms "predictable" and "controllable" leap out from the page. Unfortunately, the environment outside our organizations shapes our destiny, and it is neither predictable nor controllable. We might consider a pattern to be a "fuzzy" process -- one we can use based on successful experience, but not one for which we can explicitly identify a deterministic, cause-and-effect relationship.
Process-driven initiatives also fail to adequately distinguish between action and decisionmaking. As the glut of information assaults our organizations, decisionmaking must be driven by those who possess the best information. As information and action are distributed, decisionmaking must follow. decisionmaking, particularly distributed decisionmaking, is an art form requiring judgment, improvisation, influencing, and collaboration -- not characteristics that can be wrapped up in nice little process packets.
For these reasons -- focusing on process at the expense of people, decisionmaking, and assuming that the world is predictable and controllable -- we need a new word to express a new viewpoint. The word "pattern" provides a viable alternative. It depicts a framework for thinking, rather than a set of rules to obey. Patterns are far from a sure thing; they increase the odds, but they are not a guarantee. Growing out of the object and component technology realm, the term "pattern" has become more widely used over the past several years. Books on design patterns and analysis patterns are becoming popular; there is even a recent book entitled Cognitive Patterns that discusses frameworks for solving problems. We might consider a pattern to be a thinking person's process. Certain activities are in fact processes -- configuration control, for example. Other activities are better described as patterns -- requirements analysis comes to mind.
This article is not intended to be a treatise on Leadership-Collaboration management but an introduction to it. So using our new perspective on the word "pattern," L-C management will be illustrated by discussing three patterns: balancing at the edge of chaos, balancing at the edge of time, and balancing at the edge of power.
Balancing at the Edge of Chaos
First, let's get over our need to be tidy and orderly. Of course there are places where orderliness is necessary, but in general, the competitive world's pace of change precludes it. Surely we need appropriate practices such as risk management and configuration control, but to assemble more and more of these practices under the roof of orderliness and statistically controlled rigor ignores the real world. My guideline for rigor in complex IT projects is, "Employ slightly less than just enough." Why?
Complex problems -- those characterized by high speed, high change, and uncertainty -- are solved by creativity, innovation, good problem solving, and effective decisionmaking. All of these capabilities suffer from an emphasis on stabilizing processes. To stabilize a process (make it repeatable), we endeavor to restrict inputs, transfer only the necessary information between processes (for efficiency), and strive to make the transformation as algorithmic as possible. But complex problems in today's organizations require the interaction of many people, diverse information, out-of-the-box thinking, quick reaction, and, yes, rigorous activity at times. Balancing at the edge of chaos provides just enough rigor to keep us from plunging into chaos, but not enough to quell creative juices. Chaos is easy -- just do it. Stability is easy -- just follow the steps. Balancing is hard -- it requires enormous managerial and leadership talent.
Balancing at the Edge of Time
"How Buildings Learn is a masterful new synthesis that proposes that buildings adapt best when constantly refined and reshaped by their occupants, and that architects can mature from being artists of space to becoming artists of time," states the cover introduction to a wonderful book on building architecture by Stewart Brand.3 To paraphrase Brand, IT managers and software architects need to progress from being designers of structure to becoming architects of time.
If we view the world as stable and predict-able, then change will be viewed as a temporary aberration on the way from one stable place to another. If we view the world from the perspective that everything is unpredictable -- that is, complete randomness -- then we become immobilized with fear and indecision. Architects of time understand that everything does in fact change, but with different-length time cycles. Adaptive managers understand the middle ground: balancing the past and the future, separating stabilizing forces from destabilizing ones, and driving change instead of being driven by change.
For example, software architects succumb to the same illusions of stability as their building architect cousins. They think decisions about hardware, operating systems, networks, application servers, messaging hubs, languages, and more (the list is nearly endless) are decisions about putting a structure in place. They become so fixated on answering the question, "What is the best XYZ?" that they fail to consider the more important questions, "What is the change cycle of this architectural component?" and, "What is our adaptation strategy when it does change?" We barely get client-server applications in place and the Internet changes the game. We barely get a backbone ERP application installed and we have to make it easier to use, distribute its functionality across the Web, and integrate it with multiple other applications. Architects of time understand that the more turbulent and quick-changing the environment, the shorter the iterative management cycles need to be.
We create applications once, but we adapt them continuously. That adaptation consists of enhancement, migration, and integration. We enhance business functionality, we migrate from one technology to the next, and we integrate across applications. Many of the application integration consultants and vendors bandy about the figure of 35 percent of application delivery dollars spent on integration as an awful number, one that we can surely reduce. But maybe the percentage is actually too low. If we approach architecture from the perspective of time rather than structure, maybe the amount spent on anticipating and building architectures for enhancement, migration, and integration should be higher than 35 percent.
Architects of time balance learning from the past with anticipating future scenarios while operating predominantly in the present. A number of object technology proponents advocate throwing away the lessons from structured analysis and design, information engineering, and other past practices. But forgetting the past is silly, just as dwelling on it is. Brown and Eisenhardt offer strategies for becoming architects of time such as: "Blend the best of the past with something that is new. Carry a critical mass of experienced people forward. Probe the future with a wide variety of probes across multiple time horizons." Organizations hope that up-front software design work will translate to less rework later on; however, more design work up front may delay a critical business initiative. The same is true of architecture and analysis work. "How much is enough" is as much an issue of the business value of time as it is of traditional quality measures.
Architects of time understanding pacing. The team that drives the pace often wins in competitive sports (soccer, tennis, basketball, and football, for example). Two generic types of basketball teams are half-court teams (e.g., the Utah Jazz) and fast-break teams (e.g., the Los Angeles Lakers). Teams try to bend opponents to their own preferred style of play -- under 100 points indicates a slower game that the Jazz have a better chance of winning; over 100 points indicates a faster game that would favor the Lakers. "Generally speaking, time pacing is about creating an internal rhythm that drives the momentum for change," write Brown and Eisenhardt. "Time pacing is one of the least-understood facets of strategy in unpredictable and high-velocity industries."
Balancing at the Edge of Power
Organizational power structures impact adaptability. With all the emphasis over the last decade or so on the use of distributed teams, communities of practice, and collaboration, we have shied away from discussions about power structures in organizations. There have, however, been reams of material written and millions of consultant dollars spent on "change management," and in my opinion much of it wasted. There is a subtle difference between adapting and changing. Adapting is outwardly focused; it depends on a free flow of information from the environment (one's market) and realistic decisions based upon both information and organizational values (goals). Many changes in organizations are in response to internal politics rather than external information. Adaptive organizations learn from the outside in rather than from the inside out.
How many times have we heard the comment, "If top management doesn't get involved, XYZ won't happen"? In turbulent, high-speed markets, waiting for top management to get involved is a recipe for failure. Adaptation needs to happen at all levels, which means free-flowing information and distributed decisionmaking. It means that the traditional hierarchical management structure must share power with the horizontal, networked-team structure. In other words, there is a vertical, hierarchical power structure and a horizontal, network power structure, and an organization's capacity to adapt depends on the balance between the two. Hierarchical power is stabilizing; networked power is more adapting. Too much of the former breeds stagnation; too much of the latter breeds chaos. Organizations with rigid hierarchical power structures might just as well save their change management consulting dollars. Sharing power is a fundamental characteristic of adaptable organizations.
In organizations, power defines who gets to make decisions. Over time, the pattern of decisions an organization makes determines success and failure -- for example, the right product beats the product right most of the time. According to Nobel laureate Herbert Simon, decisionmaking depends on understanding the difference between value and facts and using the proper balance of each. Simon writes, "[The book] Administrative Behavior was written on the assumption that decisionmaking processes hold the key to understanding organizations."4 Control-oriented managers hoard decisionmaking. New-age managers empower others to make decisions, often diffusing power so much that little gets done. Adaptive managers understand that making good decisions is more important than prescribing activities. Furthermore, they understand when to make decisions and when to distribute them. It is a constant, delicate balancing act that depends on understanding patterns that work rather than processes that prescribe.
"Emotion is a principle of motivation, focusing us toward particular goals," says Simon. "It is this same intensity of thought that allows us to concentrate on solving highly complex problems." It is ironic that in the era of e-business, more information results in less certainty. With so much information, much of it contradictory, we first struggle with what information our organization should regard as "fact" and then struggle with evaluating those facts based on our implicit and explicit values. Values, whether they are reflected in company mission statements or project objectives, are critical in both separating fact from fantasy and in making choices once those "facts" have been determined.
Values are emotional. If we aren't pass-ionate about our values, why have any? Repeatable processes assume an underlying predictability and a "fact-based" evaluation process. But we know that decisionmaking, particularly decisionmaking in times of high uncertainty, is driven by the values we use to make choices, and values are about passion. The ultimate irony may be that to the extent that rigorous processes drive out passion, we are less and less able to make difficult decisions and solve complex problems because we subvert the very passion we need to succeed.
PEERING INTO THE FUTURE
The contents of this article may seem distant from the daily crunch and grind of the everyday IT manager. But trying to peer into the next decade, much less the next century, is tough. To listen to pundits, IT management is usually behind the curve rather than in front of it. There is an innate and often justifiable conservatism in what we do. At the 1999 Cutter Consortium Summit, Steve Andriole, senior consultant with the Cutter Consortium, commented that the trend to outsource IT was accelerating. E-business (which I use as a catchall phrase to label our new economy) is changing corporate landscapes. Maybe the trouble we are having getting out in front on issues like e-business is in part because of our long history of focusing on optimizing management practices. Make no mistake, many of these practices are still valuable and necessary -- just insufficient. However, it seems to me that a fundamental shift is in order. This shift will affect how we manage IT organizations as a whole, how we manage projects, and how we deliver new software.
The only prediction I will make about the future is that it will be
unpredictable. If that is true, then our emphasis on optimizing management
practices, on repeatable processes and step-wise maturity, is doomed to
fail. Getting IT in front of the curve means altering our fundamental
premise from one of optimization to one of adaptation, moving from
Command-Control processes to Leadership-Collaboration patterns, and
finding better ideas. As authors Howard Sherman and Ron Schultz5
state so well, "Our businesses are only as successful as our ideas.
If your ideas are out of date, the behaviors they drive will be out of
Jim Highsmith is president of Information Architects, Inc. and editor
of e-business Application Delivery, the monthly newsletter on
developing and delivering applications in an e-business economy. Mr.
Highsmith has 30 years' experience as a consultant, software developer,
and manager. His experience spans several IT disciplines, including
technology planning, software engineering, project management, and
performance improvement. Mr. Highsmith has held technical and managerial
positions in the computer industry with both a CASE vendor and a hardware
vendor. He has also held technical and management positions in the banking
and energy industries.
Mr. Highsmith is a regular speaker at technology conferences in the US
and abroad. In addition to speaking and consulting, Mr. Highsmith is the
author of Adaptive Software Development: A Collaborative Approach to
Managing Complex Systems and the primary developer of RADical Software
Development®, a rapid development framework.
Jim Highsmith is president of Information Architects, Inc. and editor of e-business Application Delivery, the monthly newsletter on developing and delivering applications in an e-business economy. Mr. Highsmith has 30 years' experience as a consultant, software developer, and manager. His experience spans several IT disciplines, including technology planning, software engineering, project management, and performance improvement. Mr. Highsmith has held technical and managerial positions in the computer industry with both a CASE vendor and a hardware vendor. He has also held technical and management positions in the banking and energy industries.
Mr. Highsmith is a regular speaker at technology conferences in the US and abroad. In addition to speaking and consulting, Mr. Highsmith is the author of Adaptive Software Development: A Collaborative Approach to Managing Complex Systems and the primary developer of RADical Software Development®, a rapid development framework.
1This article is based on material in James A. Highsmith, Adaptive Software Development: A Collaborative Approach to Managing Complex Systems (Dorset House Publishing, 1999).
2Shona L. Brown and Kathleen M. Eisenhardt, Competing on the Edge: Strategy as Structured Chaos (Harvard Business School Press, 1998).
3Stewart Brand, How Buildings Learn: What Happens After They Are Built (Penguin Books, 1994).
4Herbert A. Simon, Administrative Behavior: A Study of Decision-Making Processes in Administrative Organizations, 4th ed., (The Free Press, 1997).
5Howard Sherman and Ron Schultz, Open Boundaries: Creating Business Innovation Through Complexity (Perseus Books, 1998). © 1999 by Jim Highsmith. All rights reserved.
© Copyright 1999 Cutter Information Corp. All rights reserved. No part of this document may be reproduced in any manner without express written permission from Cutter Information Corp.