Paul Gerrard discusses the future of testing and automation in an environment focused on digital transformation and continuous delivery. He argues that the traditional testing models are no longer relevant and proposes a new model of testing focused on exploration, judgment, and building test models from various sources of knowledge. Under this new model, all testing is seen as exploratory in nature. Gerrard also emphasizes the importance of shifting testing activities left in the development process through early collaboration to help address issues with requirements. Automation is framed as only one part of the overall testing process and trust in automation requires proactive efforts to reduce doubts through addressing underlying issues identified earlier in development.

1. Automating Assurance:
Tools, Collaboration and DevOps
Paul Gerrard
paul@gerrardconsulting.com
Twitter: @paul_gerrard
Web: gerrardconsulting.com
Slide 1

2. AreYou Digital?
Digital IS IT

3. "Automation is the future!"
• Heard that before?
• What exactly is possible and impossible with automation, right
here, right now?
• Where are the Digital, DevOps and Continuous Delivery crazes
leading us?
• Where do testers fit?
• How do testers work in high-paced, automation-dominated
environments?
• Let's look into the near future and discuss how we survive or
thrive.
Slide 3

4. Digital is a Business Initiative
Agile (an IT initiative) has taken
17 years to get half way
Digital – it’s already there in most places

5. What is Digital anyway?
• Digital is the buzz-phrase of the moment
– Social, Mobile,Analytics, Cloud (SMAC)
– Consumerization of IT
– Buzzword bingo*
• Agile laggards (gov, large companies) playing catch-up
• Customer Experience Optimization (CXO) is critical now
• S/W development at the pace of marketing.
* https://zoomph.com/blog/top-20-digital-marketing-buzzwords/
Slide 5

6. Frequent/regular s/w delivery is critical
• Mobile users expect apps to change almost daily
– New features, offers, opportunities all the time
• Users use the best apps to get what they want
– Not necessarily the best supplier
• Businesses are in an APPS RACE
• Speed of delivery is...
– Partly about pro-action
– Partly about survival.
Slide 6

7. Automation (not just test) is critical
• Business wants IT responsiveness (true agility)
– Not necessarily 100s of releases/day
– Rapid, regular turnaround from ideas to software delivery
• With continuous integration/deployment, DevOps, developers
can now promise Continuous Delivery
• Testers need to provide Continuous Assurance
• “Test automation through the (shortened) lifecycle"
• But test automation success is still hard to achieve.
Slide 7

8. A recent survey
Slide 8
? - 1997

9. What Goes Wrong with
Test Execution Automation?

10. (GUI) Tools are sensitive instruments
Slide 10
You don't need tools and tools won't help, if your software is unstable.

11. The anti-regression goal
• What are you trying to do with automation?
• Anti-regression is the primary goal
– Detect unwanted change (functional equivalence)
– Insurance that developers can make changes safely
• How do you achieve this?
– The automation model must align with the developer model
– You have to collaborate early to achieve this.
Slide 11

12. Testing and automation – a
modelling problem not a tools
problem
Need to shift testing thinking to the left.
We model to improve requirements,
to eliminate mistakes in code and automation.

13. Brian Marick's Original 2003 Agile Testing
Quadrants
Slide 13

14. Slide 14
Crispin and Gregory(Agile Testing version)

15. Slide 15
The V-Model
appeared
in the 1980s

16. Slide 16

17. What does this tell you?

18. Are these models useful and still
relevant?
I don’t think so

19. Slide 19

20. Slide 20

21. Questions arising
• Automation is still (after 30+ years) a challenge
• We haven't been able to implement tools as effectively as we
would like to
• What are the prospects of getting better at test automation?
• What are the alternatives?
• Is there a cultural challenge too?
• Will testing and testers be the barrier to success in Digital?
Slide 21

22. The old ways won't work in the
future
We need a New Model ofTesting (free from
logistics)
Tools don’t think; they support our thinking

23. Forget Logistics
(for the time being)
Document or not?
Automated or manual?
Agile v waterfall?
This business or that business?
This technology v that technology?
Quadrants, pyramids and ice-cream cones?

24. ALLTesting is Exploratory
We explore sources of knowledge ...
... to build test models ...
... that inform our testing.

25. Judgement, exploring and testing
Testing
(the system)
Our model(s) are adequate
Our model(s) are not adequate
Exploring
(sources)
Judgement
Creates test
models
Uses test
models
We explore sources of knowledge to build test models that inform our testing
We don't yet know what the
system should do.
We can't test yet
(learning what it should do)
We believe that we know what
the system should do.
We can (automate) test.
(learning what it does do) Slide 25

26. Judgement, exploring and testing
Slide 26
Testing
(the system)
Our model(s) are adequate
Our model(s) are not adequate
Exploring
(sources)
Judgement
Creates test
models
Uses test
models
We explore sources of knowledge to build test models that inform our testing
BTW – Do Developers explore the same way? I think so.

27. Exploration process
Slide 27
ExplorationDefinitions
specs/stories
People
(& you)
Sources
Require-
ments
Test
Models
Enquiring
Challenging
Sources:
People, documents,
experience, system under test
Modelling
Test Models:
Can be documented
or mental models
Predicting
System
under test

28. Testing process
Slide 28
Testing
System
Under Test
Refining
Informing Applying
Interpreting
Test
Models
Revise the
System
More exploring Reporting

29. New ModelTesting
90 Minute talk: https://www.youtube.com/watch?v=G7xAhPACyso (go to 23m 05s)
29 page paper: http://dev.sp.qa/download/newModel Slide 29

30. The term Test Automation misleads
• It misleads as a label because the whole of testing cannot be
automated
• The label is bad, but the scope ofTest Automation is what I call
‘Applying’ in the New Model of Testing (1 process in 10)
Checking fits here (if
you’re interested)
Slide 30

31. Testers, DevOps and
Continuous Delivery

32. New ModelTesting
Slide 32
The Left The Right

33. Left and Right
Slide 33
Automation won’t
work if the thinking to
the left is not in place
Thinking on the left
contributes to the definition
of features and how you test
them
If you aren’t involved on the left,
you aren’t in a good position to
test (with or without tools)
Models on the left are the
blueprints for the tests &
automation on the right
Your opportunity to
flush out muddled,
loose, incomplete,
flaky requirements
is on the left
Applying automation to muddled,
loose, incomplete, flaky features is
doomed before you start
The Left The Right

34. Trust

35. Tests,Automation andTrust
• Can we trust automation to do all our testing?
• Consider four areas:
– Checks that can be automated by developers
– Checks that can be automated (typically by system testers) to
exercise API-level, end-to-end
– Compatibility checks across browsers, operating systems, platforms.
– Tests that can only be performed by a human
• Can you 'let go' of late, manual checking?
• It requires proactive effort and trust to reduce doubt.
Slide 35

36. Shifting left/redistributing responsibility
(AKA Test Strategy)
• Story-boarding, BDD stories, prototyping – constantly
challenging through examples
• Push more testing responsibility to the developers
• Give support to developers (risks, test ideas, challenge)
• Automate unit/integration tests + manual exploratory tests
during development
• Reduce dependence on automated end-to-end/UI tests
• Highly selective UI tests used for regression/automation.
Slide 36

37. How will shifting left reduce doubt?
• Testers must become pathfinders: identify/assess risks, formulate
tests; ensure they are included in automation etc.
• You are not:
– The gatekeeper of quality
– The last defence
– The only person who cares
• Think and act like a visionary, customer advocate, risk-identifier and
risk manager, a facilitator and a coach/mentor (and test a bit too).
Slide 37

38. The Future is CloserThanYou
think

39. The Need for Change
82% of teams say they are not meeting
their users’ expectations and achieving
their desired business outcomes
89% of teams say they can’t keep up with
the pace of DevOps
56% of teams say that they can’t automate
testing for significant parts of their App
72% of teams say they lack visibility of their
quality status; 98% lack confidence in their
ability to predict the impact of changes
78% of teams say they can’t find people
with the technical skills and the domain
knowledge that they need
Antony Edwards, CTO Eggplant
“Sources are Gartner, Forrester,
World Quality Report, Eggplant survey”

40. Modelling, test and automation (tools coming)
Features/Forms/Services System Models Test Pattern
Hierarchy
Process Flow
State Model
Feature, Form
or Unit
User, workflow,
acceptance
Integration
Slide 40

41. What’s coming soon
• CXO is the drive towards positive business outcomes
– Higher enquiry rates
– Higher registration rates
– Higher conversion rates and sales
• Real-time User Monitoring (RUM) collects data from
production and will be mapped to system release changes
• With machine learning, patterns emerge and AI will drive the
next iteration of development; enable effective impact analysis.
Slide 41

42. The testing opportunity
• You need to change, but what an opportunity!
• Testers need to change behaviours:
– To think much more analytically (modelling!)
– Move from passive to active collaboration
– Challenge and refining requirements
– Mapping change to business outcomes
– Acting like a pathfinder or navigator in projects
Slide 42
Testers need to step up to Digital Assurance

43. Automating Assurance:
Tools, Collaboration and DevOps
Thank-You!
Paul Gerrard
paul@gerrardconsulting.com
Twitter: @paul_gerrard
Web: gerrardconsulting.com
Slide 43

44. References
• TheTester’s Pocketbook
– http://testers-pocketbook.com (2011)
• Will Robots ReplaceTesters?
– https://www.youtube.com/watch?v=EzyUdJFuzlE (2016)
• Thinking Big: Introducing Test Analytics
– https://gerrardconsulting.com/blog/thinking-big-introducing-test-analytics/ (2013)
• Digital Assurance
– http://transform.ca.com/digital-assurance-book.html (2017)
• New ModelTesting:A New Process andTool
– https://gerrardconsulting.com/blog/new-model-testing-a-new-test-process-and-
tool/
Slide 44