Commercial Off The Shelf (COTS) applications are a part of most (perhaps every) medium to large enterprises. A lot of money is spent every year in selection and implementation projects of such applications – think millions to billions of dollars if you consider lost opportunity costs (which are hard to measure). Unfortunately, buying instead of building is no guarantee of success. Tales of monumental package implementation overruns and disappointing functionality – not to mention outright failures – abound. These failures occur despite elaborate, detailed evaluation processes meant to select just the right package and reduce the risk of implementation. It is clear that businesses have to get better at researching, evaluating, and selecting software products. In this talk we’ll describe an approach to selecting and implementing package software that focuses on engineering quality and business agility. We prefer an approach that involves hands-on, experiential hypothesis testing and evidence gathering over endless feature checklists and vendor demonstrations.

1. HYPOTHESIS-DRIVEN
COTS SOFTWARE SELECTION
Agile Australia 2017
@tiagogriffo

2. Principal Technologist at ThoughtWorks
Developer
Seen too many bad COTS software selection
processes
2

3. Large software projects are risky. There is an
assumption that by implementing COTS we will be
reducing the risk.
That assumption very often doesn’t hold true.
Even more often COTS gets in the way of IT
performance and bottom line results.
3

4. 4
TALES OF
MONUMENTAL
FAILURE

5. 5
http://spectrum.ieee.org/static/monuments-to-failure

6. 6

7. 7
DO WE BUILD
OR DO WE BUY?

8. 8
TRADITIONAL REASONS FOR
BUYING
Software Development is Risky
Software Development is
Inefficient
Software Development is NOT
Our Core Competency
https://erik.doernenburg.com/2012/09/buy-vs-build-shift-part-1/

9. 9
TRADITIONAL ISSUES WHEN
BUYING
Deliver Features That Are Not
Needed
Bring Their Own Roadmap
Require Customisation
https://erik.doernenburg.com/2012/09/buy-vs-build-shift-part-1/

10. BUILD OR BUY?
Utility Strategic
Better to build! This
differentiates your
company
Ok to buy! Change
the process
https://martinfowler.com/bliki/UtilityVsStrategicDichotomy.html

11. 11
THE
TRADITIONAL
APPROACH

12. GOA
Golf Oriented Architecture

13. GENERIC, PHASED TRADITIONAL
APPROACH
Planning and
Budgeting
Requirements
Gathering/
Analysis
Vendor Research
Demos and Data
Capturing
Decision and
Contract
Negotiation

14. 14
HiPPO
Highest Paid Person Opinion

15. 15
Who has been involved in a COTS software
selection process like this before?

16. 16
Keep your hands up if you think
the process achieved the results
you expected it to.

17. WE NEED TO SELECT
SOFTWARE THAT
HELPS US MOVE
TOWARDS A HIGHER
PERFORMANCE IT
ORGANISATION

18. 18
WHAT DOES HIGH IT
PERFORMANCE LOOK
LIKE

19. IT Performance
Deployment
Frequency
Deployment Lead
Time
Mean Time To
Recover
Cycle Time
MTTR over MTBF
2016 State of DevOps Report presented by Puppet and Dora

20. or in a few words:
THE ABILITY TO RESPOND AND
CHANGE QUICKLY AND SAFELY

21. How is customising COTS software
different than writing software in a
bespoke language?

22. Customisations should be
version controlled

23. 23
https://martinfowler.com/bliki/TestPyramid.html
TEST AUTOMATION

24. INFRASTRUCTURE
AUTOMATION

25. OTHER REALLY IMPORTANT STUFF THAT OFTEN
DOESN’T GET THE ATTENTION THEY DESERVE
Integration Modularisation UpgradesEnvironments

27. WHAT DOES HYPOTHESIS
DRIVEN SOFTWARE
SELECTION LOOK LIKE?

28. WE BELIEVE
the ability to apply
agile, continuous
delivery and devops
practices to our COTS
systems
WILL RESULT
in rapid evolution,
value delivery and
overall higher IT
performance
WE WILL HAVE
CONFIDENCE
to sign a contract with
the vendor when we
can make small
changes that deliver
business value in a
safe and quick way

29. 29
HYPOTHESIS-DRIVEN PROCESS
Important
Hypothesis
Biggest Risks Agile-driven Experiment
COTS A / Team A
Agile-driven Experiment
COTS B / Team B
Lessons Learned,
Compromises
COTS Decision

30. RECAP
30

31. THANK YOU