Agile Is the New Waterfall

Agile (as practiced today) is the new Waterfall Embrace Uncertainty Naresh Jain naresh@agilefaqs.com Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 1

For me to make sense lets look at... Some examples of heavy weight process Lean Thinking Origin of Agile Agile Values and Principles Summary of how Agile helps Where do things go Wrong? Naked Agile Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 2

Software Engineering? Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 3

Software Engineering? Creating Software is a Craft. Converting source code to executable is the engineering bit. Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 3

IEEE deﬁnes Software Engineering as... “Software Engineering is the application of a systematic, disciplined, quantiﬁable approach to development, operation and maintenance of software: that is, the application of engineering to software.” IEEE Standard Computer Dictionary, ISBN 1-55937-079-3, 1990 Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 4

Who used Software Engineering? Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 5

For the space shuttle’s operating system Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 6

Some Statistics NASA’s Defect Density Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 7

Some Statistics NASA’s Defect Density The last 11 versions of the space shuttle’s 420,000 line systems had a total of 17 defects. Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 7

One More Data Point Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 8

Another real software engineering project Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 9

Another real software engineering project Safeguard - Ballistic Missile Defense System Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 9

Another real software engineering project Safeguard - Ballistic Missile Defense System 18 20 code & unit test design 18 % 20 % 1969-1975, 5407 person years reqmts Hardware designed at the same time 20 % 20 integration as software specs being written testing 42 % Late changes in requirements not an option 42 Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 9

Another real software engineering project Safeguard - Ballistic Missile Defense System 18 20 code & unit test design 18 % 20 % 1969-1975, 5407 person years reqmts Hardware designed at the same time 20 % 20 integration as software specs being written testing 42 % Late changes in requirements not an option 42 Did it Succeed? Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 9

Safeguard Ballistic Missile Defense System… Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 10

Safeguard Ballistic Missile Defense System… Revised Project Statistics Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 10

Safeguard Ballistic Missile Defense System… Revised Project Statistics The project was delivered according to speciﬁcations Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 10

Safeguard Ballistic Missile Defense System… Revised Project Statistics The project was delivered according to speciﬁcations Cost: $25 Billion (not adjusted) Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 10

Safeguard Ballistic Missile Defense System… Revised Project Statistics The project was delivered according to speciﬁcations Cost: $25 Billion (not adjusted) 1969-1975, 5407 person years Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 10

Safeguard Ballistic Missile Defense System… Revised Project Statistics The project was delivered according to speciﬁcations Cost: $25 Billion (not adjusted) 1969-1975, 5407 person years Operational for 133 days - Project terminated in 1978 Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 10

Safeguard Ballistic Missile Defense System… Revised Project Statistics The project was delivered according to speciﬁcations Cost: $25 Billion (not adjusted) 1969-1975, 5407 person years Operational for 133 days - Project terminated in 1978 ‘By the time the 6-year anti-missile system project was completed, the new missiles were faster than the anti- missile missiles’ Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 10

Where did things go wrong? Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 11

Where did things go wrong? Software Engineering is a heavy weight methodology and such heavy weight methodologies characteristically are most successful when: Requirements are stable Technology is well known and mature Everything happens as one would expect We are not taking on anything new or unknown We have done this many times before Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 11

Where did things go wrong? Software Engineering is a heavy weight methodology and such heavy weight methodologies characteristically are most successful when: Requirements are stable Technology is well known and mature Everything happens as one would expect We are not taking on anything new or unknown We have done this many times before Projects with these characteristics are few and far between. Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 11

Other Heavy Weight Methodologies Heavy Weight SEI/IEEE Project Waterfall Standards and Definitions Requirements RUP Management Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 12

Other Heavy Weight Methodologies Heavy Weight SEI/IEEE Project Waterfall Standards and Definitions Requirements RUP Management Heavy weight methodologies work in some instances, but there are high costs, and the risk in using them in dynamic environments is high. Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 12

So, heavy weight methodologies don’t seem to meet our needs Is there an alternative? Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 13

Agenda for this session Some examples of heavy weight process Lean Thinking Origin of Agile Agile Values and Principles Summary of how Agile helps Where do things go wrong? Naked Agile Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 14

Lower cost of change through higher quality software Traditional cost proﬁle Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 15

Lower cost of change through higher quality software Traditional cost proﬁle Time spent: • Finding defects • Fixing Defects • Regression Testing • Deploying Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 15

Lower cost of change through higher quality software Traditional cost proﬁle Time spent: • Finding defects • Fixing Defects • Regression Testing • Deploying More defects found Agile system cost proﬁle & corrected – causes lower cost of defect correction Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 15

Lean Thinking – Eliminate Waste The Toyota Production System, 1988 (1978), Taichii Ohno Pull Scheduling - Just-in-Time Flow Expose Problems - Stop-the-Line Culture Study Of ‘Toyota’ Production System, 1981, Shigeo Shingo Non-Stock Production - Single Minute Setup Zero Inspection – Automatic Error Detection at Every Step Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 16

Building Quality into the Process Toyoda Loom Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 17

Lessons from Queuing theory Utilization (%) Source: Beyond Agile Software Development Becoming Lean, Mary Poppendieck, Poppendieck.llc Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 18

Traditional Process Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 19

Applying Lean Principles to Software Development Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 20

Applying Lean Principles to Software Development End-to-End small slices of work Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 20

Applying Lean Principles to Software Development End-to-End small slices of 20 % done = 100 % usable work Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 20

Lean Principles applied to Software Development Project Plan/Estimation Requirements Gathering Use Cases / Functional Specs Design Specifications Code Test $ Fix / Integrate Inception $ $ $ $ Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 21

New Methodologies Emerged XP FDD Scrum Crystal Family DSDM Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 23

2000 Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 24

2000 XP | Extreme Programming (Kent Beck) DSDM | Dynamic System Development Method (Dane Faulkner) FDD | Feature Driven Development (Jeff DeLuca) SCRUM (Ken Schwaber) Crystal (Alistair Cockburn) Adaptive Software Development (Jim Highsmith) Lean Software Development (Mary Poppendieck) Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 24

Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 25

2001 Agile manifesto Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 25

2001 “We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: Agile manifesto Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 25

2001 “We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: Agile – Individuals and interactions OVER processes and tools. manifesto Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 25

2001 “We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: Agile – Individuals and interactions OVER processes and tools. manifesto – Working software OVER comprehensive documentation. Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 25

2001 “We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: Agile – Individuals and interactions OVER processes and tools. manifesto – Working software OVER comprehensive documentation. – Customer collaboration OVER contract negotiation. Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 25

2001 “We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: Agile Individuals and interactions OVER processes and tools. – manifesto Working software OVER comprehensive documentation. – Customer collaboration OVER contract negotiation. – Responding to change OVER following a plan. – Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 25

2001 “We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: Agile Individuals and interactions OVER processes and tools. – manifesto Working software OVER comprehensive documentation. – Customer collaboration OVER contract negotiation. – Responding to change OVER following a plan. – That is, while there is value in the items on the right, we value the items on the left more.” © 2001 Agile Alliance. http://www.agilemanifesto.org Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 25

Principles behind the Manifesto Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. Business people and developers must work together daily throughout the project. Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 27

Principles behind the Manifesto... Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. The most efﬁcient and effective method of conveying information to and within a development team is face-to-face conversation. Working software is the primary measure of progress. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indeﬁnitely. Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 28

Principles behind the Manifesto... Simplicity--the art of maximizing the amount of work not done--is essential. Continuous attention to technical excellence and good design enhances agility. The best architectures, requirements, and designs emerge from self-organizing teams. At regular intervals, the team reﬂects on how to become more effective, then tunes and adjusts its behavior accordingly. Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 29

Agile is Designed to deal with Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 30

Agile is Designed to deal with Ziv's law - speciﬁcations will never be fully understood. Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 30

Agile is Designed to deal with Ziv's law - speciﬁcations will never be fully understood. Humphrey's law - the user will never know what they want until after the system is in production (maybe not even then) Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 30

Agile is Designed to deal with Ziv's law - speciﬁcations will never be fully understood. Humphrey's law - the user will never know what they want until after the system is in production (maybe not even then) Wegner's lemma - an interactive system can never be fully speciﬁed nor can it ever be fully tested. Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 30

Agile is Designed to deal with Ziv's law - speciﬁcations will never be fully understood. Humphrey's law - the user will never know what they want until after the system is in production (maybe not even then) Wegner's lemma - an interactive system can never be fully speciﬁed nor can it ever be fully tested. Langdon's lemma - software evolves more rapidly as it approaches chaotic regions (taking care not to spill over into chaos) Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 30

Agile is Designed to deal with Ziv's law - specifications will never be fully understood. Humphrey's law - the user will never know what they want until after the system is in production (maybe not even then) Wegner's lemma - an interactive system can never be fully specified nor can it ever be fully tested. Langdon's lemma - software evolves more rapidly as it approaches chaotic regions (taking care not to spill over into chaos) Any association of predictive or defined processes with Agile is an exercise in futility. - Jeff Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 30

Use of Agile Methods Helps handle changing requirements & priorities Lowers cost of change Provides better visibility into project progress Reduces risk Maximizes Return on Investment (business value prioritized) Encourages higher quality, simpler code Delivers business value early & often Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 32

But...with.... Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 33

But...with.... Courage!! Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 33

But...with.... Courage!! Constant Business involvement Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 33

But...with.... Courage!! Constant Business involvement A need for more Discipline Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 33

But...with.... Courage!! Constant Business involvement A need for more Discipline Greater emphasis on Testing Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 33

But...with.... Courage!! Constant Business involvement A need for more Discipline Greater emphasis on Testing Whole Organization Involvement Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 33

But...with.... Courage!! Constant Business involvement A need for more Discipline Greater emphasis on Testing Whole Organization Involvement Last but not the least, with an OPEN MIND Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 33

So what makes Agile the new Waterfall? Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 35

So what makes Agile the new Waterfall? Do you really think this is how companies are adopting Agile? Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 35

So what makes Agile the new Waterfall? Do you really think this is how companies are adopting Agile? Mass adoption of Scrum/XP across the organization Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 35

So what makes Agile the new Waterfall? Do you really think this is how companies are adopting Agile? Mass adoption of Scrum/XP across the organization What about Organizational Transformation & Cultural Changes? Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 35

So what makes Agile the new Waterfall? Do you really think this is how companies are adopting Agile? Mass adoption of Scrum/XP across the organization What about Organizational Transformation & Cultural Changes? Unreasonable, false expectations Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 35

So what makes Agile the new Waterfall? Do you really think this is how companies are adopting Agile? Mass adoption of Scrum/XP across the organization What about Organizational Transformation & Cultural Changes? Unreasonable, false expectations Making Agile a very prescriptive, consistent PROCESS Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 35

So what makes Agile the new Waterfall? Do you really think this is how companies are adopting Agile? Mass adoption of Scrum/XP across the organization What about Organizational Transformation & Cultural Changes? Unreasonable, false expectations Making Agile a very prescriptive, consistent PROCESS Process and Tools OVER Teams and their needs Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 35

So what makes Agile the new Waterfall? Do you really think this is how companies are adopting Agile? Mass adoption of Scrum/XP across the organization What about Organizational Transformation & Cultural Changes? Unreasonable, false expectations Making Agile a very prescriptive, consistent PROCESS Process and Tools OVER Teams and their needs Scrum Masters and Coaches are the new Process Police (Silver Bullet) Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 35

So what makes Agile the new Waterfall? Do you really think this is how companies are adopting Agile? Mass adoption of Scrum/XP across the organization What about Organizational Transformation & Cultural Changes? Unreasonable, false expectations Making Agile a very prescriptive, consistent PROCESS Process and Tools OVER Teams and their needs Scrum Masters and Coaches are the new Process Police (Silver Bullet) Business model built around Agile Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 35

So what makes Agile the new Waterfall? Do you really think this is how companies are adopting Agile? Mass adoption of Scrum/XP across the organization What about Organizational Transformation & Cultural Changes? Unreasonable, false expectations Making Agile a very prescriptive, consistent PROCESS Process and Tools OVER Teams and their needs Scrum Masters and Coaches are the new Process Police (Silver Bullet) Business model built around Agile Certiﬁcation Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 35

So what makes Agile the new Waterfall? Do you really think this is how companies are adopting Agile? Mass adoption of Scrum/XP across the organization What about Organizational Transformation & Cultural Changes? Unreasonable, false expectations Making Agile a very prescriptive, consistent PROCESS Process and Tools OVER Teams and their needs Scrum Masters and Coaches are the new Process Police (Silver Bullet) Business model built around Agile Certiﬁcation Tools vendors Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 35

So what makes Agile the new Waterfall? Do you really think this is how companies are adopting Agile? Mass adoption of Scrum/XP across the organization What about Organizational Transformation & Cultural Changes? Unreasonable, false expectations Making Agile a very prescriptive, consistent PROCESS Process and Tools OVER Teams and their needs Scrum Masters and Coaches are the new Process Police (Silver Bullet) Business model built around Agile Certiﬁcation Tools vendors Training and Coaching by non-practitioners Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 35

So what makes Agile the new Waterfall? Do you really think this is how companies are adopting Agile? Mass adoption of Scrum/XP across the organization What about Organizational Transformation & Cultural Changes? Unreasonable, false expectations Making Agile a very prescriptive, consistent PROCESS Today Agile seems to be full oftheir needs Process and Tools OVER Teams and Ceremony and Dogmatism Scrum Masters and Coaches are the new Process Police (Silver Bullet) Business model built around Agile Certiﬁcation Tools vendors Training and Coaching by non-practitioners Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 35

Where do things go wrong? Lack of understanding of software development Software development is not a Manufacturing process an Engineering or Construction process Ziv, Humphrey, Wegner's laws and Langdon's lemma Mythical man month Gold rush Lack of theoretical understanding of the underlying principles Myths and Fallacies Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 36

Theory to understand why Agile works Theory of Constraints and Lean Thinking Complex adaptive systems: the science of uncertainty Cognitive science: the nature of human decision making Evolutionary psychology & anthropology: the origins of social interaction & its nature Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 37

Cyneﬁn Framework Dave Snowden, Cognitive Edge Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 38

Project Design based on Cyneﬁn Dave Snowden, Cognitive Edge Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 39

Introducing “Naked Agile” Naked Agile: True essence of Agile without the Process Crap (ceremony and dogmatism) The Bloat Effect Iterations and Time-boxes are high ceremony Cease Inspection; Who needs a separate QA Team Requirements Considered Harmful Estimations Considered Harmful Projecting Velocity is Useless Simple Design;Yes Simple not Simply Design & Coding is NOT Programming TDD is not about Testing & TDD is not limited to Unit-tests Agile != Rapid Software Development; Embrace Simplicity (Art of maximizing work not done) Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 41

Questions? Thank You naresh@agilefaqs.com Licensed Under Creative Commons by Naresh Jain Thursday, April 30, 2009 42