2. Session Code: ARC206 Managing complexity in a Software plus Services World Amanda Jackson Fronde Systems Group Ltd
3. Presentation Outline (hidden slide): Speaker instructions: Complete this slide to assist your SME (subject matter expert) in evaluating your presentation flow, topic coverage, demo integration and alignment of content to your session description and level. Title: Managing Complexity in a Software plus Services World Technical Level: All levels Intended Audience: Developers, Enterprise Architects Objectives (what do you want the audience to take away from this session): 1. Understand what EA is and how things can go wrong 2. Understand complexity and how a system can end up being complex 3. Learn how to manage complexity and be successful using SIP Presentation Outline (including demos):
5. Amanda Jackson Consultant/Developer with Fronde Systems Ltd Started NZ Girl Geek Dinners in 2007 In the past has covered most roles in IT including: QA, Network Admin, Source Control Manager, Build Management, Games Development, Change Management
6. What is this talk about? Simplifying architecture through: Modeling complexity Partitioning complex systems By simplifying architecture we can more easily: Add business value Benefit from trends like Software plus Services
7. Software plus Services Combines Internet services with client and server software Increasingly complex, globally distributed systems More choices in the ways to access and manage software A new way of thinking is required at the EA level!
8. Simplify & Redefine Before we can simplify and redefine we need to understand: What Enterprise Architecture is Why we need Enterprise Architecture What can go wrong How things fail
9. What is Enterprise Architecture? Documentation or processes that describe: Structure Organization Behaviours Planning Changes
10. Enterprise Architecture “An enterprise architecture is a description of the goals of an organization, how these goals are realized by business processes, and how these business processes can be better served through technology.” Roger Sessions - Simple Architectures for Complex Organizations
11. Benefits of a good Architecture Adds value to an organization Is concerned with all aspects of an organization Ensures a good ROI Helps ensure successful projects
12. Why we need EA Based on a number of studies done in the last 10 years (such as The KPMG Canada Study, The Chaos Report and others), it has been established that more than 50% of IT projects fail outright! Andy Blumenthal – Director of Enterprise Architecture & IT Governance for the US Coast Guard
13. What can go wrong? Failed projects Loss of income
14. Why EA fails Excessive complexity Processes are: Expensive Time consuming Poorly defined Results are: Not testable Not reproducible Not directly usable Value is not delivered
15. EA Failures Four years ago the government announced to a grateful NHS a national IT program to become the world’s largest civil computer scheme. But after a breathless start, delivery dates for key software were missed, the full costs of implementation have always been unclear, and experts are divided over whether the scheme is too ambitious to ever work as originally planned. Questions that need to be answered on NHS IT plan by Tony Collins, April 2006 in ComputerWeekly,com
16. What does this mean for EA? In October 2007, Gartner predicted that 40% of all existing Enterprise Architecture programs will be shut down by 2010. Fewer than 5% of firms use Enterprise Architecture effectively. Ross, Weill & Robertson – Enterprise Architecture as a Strategy.
18. Existing Methodologies Influenced by Object Oriented design & analysis Pre-date Software-plus-Services Continuously evolving but not catching up! Do not deal well with the new S+S design model
19. The new Enterprise Architecture Focus on complexity at the enterprise level Understand the mathematics of complexity Create a model for complexity Define a process for controlling complexity Test solutions against that model
20. What is Complexity? A function of the number of states that system can find itself in!
21. Software System Complexity values (heads, tails) penny; penny = read (penny_sensor); if (penny == heads) message (“Penny is heads”); if (penny == tails) message (“Penny is tails”); end; Complexity = States Per VariableVariables
22. Prepare Penny/Heads Memo Penny = Heads Memo RE: Penny Outcome: Heads Read Penny Prepare Penny/Tails Memo Penny = Tails Memo RE: Penny Outcome: Tails Business System Complexity Complexity = Branches per Decision Point Decision Point
23. Software & Business Comparison When considering complexity we can see that: Variables are like decision points States per variable are like branches per point
25. Partitioning The most important complexity control strategy Divide and conquer to reduce complexity Reduced complexity = improved efficiency
26. The Power of Partitioning Number of autonomous systems: 1 Number of states per variable: 6 Number of variables: 12 Number of states: 2,176,782,336 Number of autonomous systems: 2 Number of states per variable: 6 Number of variables: 6 Number of states: 93,312 Reduction in complexity: 99.57%
27. Looking at it another way… 1 Bucket / 2 Dice a.k.a. 1 autonomous system 2 Buckets / 1 Die Each a.k.a. 2 autonomous systems 1 Die = 6 states 1 Die = 6 states 1 Die = 6 states 2 Die = 6 6 states Total states this bucket: 36 Total states for 2 buckets: 12
28. And now with 12 variables… 1 Bucket / 12 Dice 2 Buckets / 6 Die Each 3 Buckets / 4 Die Each 1 Die = 6 states 12 Die = 612 Total states: 2,176,782,336 66 = 46,656 states per bucket 64 = 1,296 states per bucket Total states: 93,312 Total states: 3,888
29. What is a partition? A set of subsets that divide a larger set All elements of the larger set live in one, and only one, of the subsets
30. Five Laws of Partitions Must be true partitions Definitions must be appropriate to the problem at hand The number of subsets must be appropriate The size of the subsets must be roughly equal The interactions between subsets must be minimal and well defined
31. Set 1 Set 2 Set 3 Set 4 Set 5 Which are partitions? Unpartitioned
32. What do we partition? Consider same-category-as
34. Our new partitions Food cereal soft drink flour lollies Stationery pen pencil notebook Reading newspaper Kitchenware knife cup
35. SIP (very brief) Overview SIP describes the main approaches used for: Complexity control Partitioning Simplification Iterative Delivery Overall goal is to align the IT and Business processes so they work together effectively
36. Partition Identification Start at the highest possible view of the enterprise then attempt to partition If partition is successful – repeat Keep repeating until you cannot partition any more At each stage of partitioning, assign types and deployment information
37. Start at the highest possible view Retail Organization Operations Internet Sales Retail Sales Catalogue Sales Brand Awareness Product Awareness Planning
42. Software plus Services Hugely flexible Integrate traditional software with distributed services Or not… Put services in a cloud for direct user access A dizzying, and very impressive array of S+S services The ability for the architecture to get hugely bloated and complex if not planned well – a.k.a. Bloatware!
43. How do we partition S+S? Keep It Simple Architect the simplest possible complexity Make it obvious which elements live where Make it easy to see where partitions lay Don’t repeat elements
44. How do we partition S+S? Start at the highest possible view Create a high level architecture Show each service and application as they relate to each other Show clearly defined links
45. How do we partition S+S? Architect and partition each solution Only those relevant to your current objective Make each architecture as simple as possible Don’t repeat elements in multiple architectures Result? It’s easier to see solution links Result? It’s easier to redesign the whole
46. Case Study in Complexity Complexity can creep into even the most extensively planned project Unchecked complexity leads to project failure Using Simple Iterative Partitions might have saved this project, even when it was well into failure mode
47. NPfIT Launched in June 2002 Automate and centralize NHS record keeping Automation of all patient care information Access to any patient record by any authorized health care professional in the UK Ability to book appointments with any health care facility in the UK Automation of prescription services
48. NPfIT Functionality Regional Clinical Information Systems Initial cost - approximately NZ$14 billion Infrastructure Systems Initial cost – approximately NZ$3.2 billion Shared Applications Initial cost – approximately NZ$425 million
49. NPfIT Overview Multi-billion dollar project Contracts split between at least a dozen vendors Covers a geographic area of close to 100,000 square miles Offers services to 60 million people Expected to process 300 transactions per second
50. NPfIT Architecture Overview Region1 CIS H H H H H H H H H H H Region2 CIS Region5 CIS H H CRS H H Central Apps C&B ETP H H H H H H H Region3 CIS Region4 CIS H H Contact PACS H H
51. Current status of NPfIT In crisis almost from the first day One year into the project most vendors were having problems relating to each other The worst off, the Accenture/iSoft partnership.
52. Accenture/iSoft By Sept 2006 Accenture had enough and abandoned the project They walked away from over $5 billion in revenues They wrote off in excess of $700 million they had already spent They agreed to pay over $100 million to settle legal obligations.
53. Overall Cost of NPfIT? Estimated cost range is $68 to $142 billion
54. Simple Iterative Partitions Phase 1 – audit of organization readiness Phase 2 – working on the partitioning Phase 3 – simplify the partitions Phase 4 – prioritize subsets in the partitions Phase 5 - iteration
55. Phase 1 Highlights deep distrust between NHS factions Delivers extensive training in the nature of complexity Focuses on managing complexity as the absolute highest priority
56. Phase 2 Already considerable effort made to partition However, partitions were wrong FIVE different implementations of the same, very complex system This was done on purpose due to a highly suspect business rule SIP forces removal of the dodgy business rule, reduces complexity by 80% Create high level partitions
57. Phase 3 Simplify partitions further Remove up to 90% complexity from the 20% remaining Remove unnecessary functionality Make sure all technical requirements can be traced back to a business requirement
58. Summary Complexity of business processes is linked to decision points and the paths created by those decision points Complexity of software systems is linked to the number of variables and the number of significant values those variables can take Partitioning is the single major factor in reducing complexity
59. Roger Sessions Roger Sessions, author of ‘Simple Architectures for Complex Enterprises’, has been an invaluable source of help and information whilst preparing this talk. He very kindly allowed me to use material from his book as part of this presentation. Roger is the CEO of ObjectWatch, the creator of SIP and an Enterprise Architecture expert.
Now some of you are probably thinking that I’ve made a mistake there and that I meant Documentation AND processes, but no. When talking about Enterprise Architecture as a documentation model, I’m referring to an EA that uses documents to describe the structure and behaviour of an enterprise and its information systems. With EA the process, this is where we set up a process that describes an enterprise and its information systems, and plans changes to improve the integrity and flexibility of the enterprise. Admittedly there may be documentation involved when setting up a process but documentation and processes are different things.That’s all very theoretical but still doesn’t give us a definitive definition. So how about this?
Do we therefore need to redefine what Enterprise Architecture is? Should we start from scratch and throw out everything we already know? No. There is a lot of value in the original EA methodologies but we need to start thinking with a view to the new styles of development and growing and improving upon them. Software plus Services wasn’t around when TOGAF, Zachman, Gartner and the rest of the old school enterprise architecture methodologies were dreamed up. Object Oriented development is quite neat compared to Software plus Services. You create an application, it’s self contained, you install it, that’s it. You may in the future create new releases and add new sections to the software but it’s still a self contained application. Software plus Services has endless room to grow and expand, it’s not limited to a single executable that you develop and install onto a machine. We’re reaching out into the cloud, interfaces and interactions need to be clearly defined before we start to develop solutions.