This talk suggests how we might make sense of the tools landscape of the near future, where the pressure to modernise processes and automate is greatest, and what a new test process supported by tools might look like. Takeaways: - We need to take machine learning in testing seriously, but it won’t be taking our jobs just yet - We don’t need more test automation tools; today we need tools that capture tester knowledge - Tools that that learn and think can’t work for testers until we solve the knowledge capture challenge. View On-Demand Webinar: https://youtu.be/EzyUdJFuzlE

1. Will Robots ReplaceTesters?
@paul_gerrard
Paul Gerrard
paul@gerrardconsulting.com
gerrardconsulting.com

2. Summary
• Impact and influence of tools and bots on
testing is increasing
• What is the direction of travel for tools/bots?
• Will the way we test be transformed?
• Do we need to prepare for a traumatic
change?
• How will tools and automation support
testing (or potentially replace) testers?
This session is based on “The Future of Tools in Testing”:
https://tkbase.com/resources/viewResource/14
Intelligent Definition and Assurance Slide 2

3. Software world goes “bot mad”
• Many jobs in the next ten to twenty years will
be done by bots and those jobs will effectively
disappear as career choices
• Some talk of testers being replaced by bots
and tools
• The common response:“Impossible!”
• I’m not so sure anymore
• Let’s explore what tools can do for us in a
different way than you may be used to.
Intelligent Definition and Assurance Slide 3

4. Robots won’t replace testers for
some time
• My thesis: new tools that support exploring,
thinking, recording and reporting will emerge
• Is the destination intelligent robot testers?
• The next steps we take will not require
sophisticated AI or Deep/Machine Learning
– Our goals with tools will change
– Different goals force a change of thinking and culture
• NextGen tools will pave the way for AI/ML
• I am building one.
Intelligent Definition and Assurance Slide 4
From now on, Ill use the term
Machine Learning or ML to
refer to AI and Deep Learning.

5. A milestone in human achievement?
• In March 2016, a
computer beat the best
human player of Go for
the first time
• Google’s AlphaGo
program beat Lee Sedol
the greatest living player,
by four games to one.
Intelligent Definition and Assurance Slide 5

6. Rules of Go
• Rule 1 (the rule of liberty)
Every stone remaining on the board must have at least one open
"point" (an intersection, called a "liberty") directly next to it (up,
down, left, or right), or must be part of a connected group that has at
least one such open point ("liberty") next to it. Stones or groups of
stones which lose their last liberty are removed from the board.
• Rule 2 (the "ko rule")
The stones on the board must never repeat a previous position of
stones. Moves which would do so are forbidden, and thus only moves
elsewhere on the board are permitted that turn.
• All other information about the game is heuristic –
learned through experience of play
• Chess: 10120 possible moves
• Go: 10761 possible moves a mere 10641 times as many.
Intelligent Definition and Assurance Slide 6

7. Why is AlphaGo significant?
• There is no possibility of computing all (or even
the next few) Go moves by computer
• Humans recognise patterns, play by intuition and
imagination
• Is AlphaGo simulating human intuition and
imagination?
• Like Go, testing is simple in theory, but is highly
complex in practice
• Could testing be computerised in the same way?
Intelligent Definition and Assurance Slide 7

8. A recent study*…
• Over the next two decades, 47% of jobs in the
US may be under threat
• It ranks 702 occupations in order of their
probability of computerisation
– Telemarketers: 99% likely
– Recreational therapists: 0.28% likely
– Computer programmers: 48% likely
• Something significant is going on out there
– If programmers have a 50/50 chance of being replaced
by robots, we should think seriously about how the
same might happen to testers.
Intelligent Definition and Assurance Slide 8
* “The future of employment: how susceptible are jobs to computerisation?”
http://www.oxfordmartin.ox.ac.uk/downloads/academic/The_Future_of_Employment.pdf

9. Some systems-related occupations
Intelligent Definition and Assurance Slide 9
Occupation Rank (out of
702)
Probability of
Computeris-
ation
Computer and Information Research Scientists 69 1.5%
Network and Computer Systems Administrators 109 3.0%
Computer and Information Systems Managers 118 3.5%
Information Security Analysts, Web Developers,
and Computer Network Architects
208 21%
Computer Occupations, All Other 212 22%
Computer Programmers 293 48%
Computer Support Specialists 359 65%
Computer Operators 428 78%
Inspectors, Testers, Sorters, Samplers and
Weighers
670 98%

10. Some observations
• The ‘robots are coming’ meme implies that it is ML
that is the driver for all this
– Much of this is hype, with the industry trying to sell the
next big thing to business
– Nothing new there
• Often, there is little or no need for ML
– Inspectors, testers and telesales are likely to be replaced
by sensors and data collectors in factories or Interactive
Voice Response (IVR) systems
– Data is larger and analysed in more sophisticated ways
– The human interaction in those occupations isn’t
sophisticated.
Intelligent Definition and Assurance Slide 10

11. Test Automation =
Mechanical Tools
What we REALLY need are
THINKING TOOLS

12. Intelligent Definition and Assurance
The term Test Automation misleads
• It misleads as a label because the whole of
testing cannot be automated
• The label is bad, but the scope of Test
Automation is what I call ‘Applying’ in the
New Model of Testing
Slide 12

13. Test Automation: MechanicalTools
• Test execution tools have been around since the 1970s
• Other tools in this category are those which perform
logistical or practical tasks:
– Creation and management of environments and data
– Test harnesses
– Mocking
– Set-up, tear-down and clean-up
• These tasks have always been part of the test execution
process
• Modern tools are slicker but these tools have not evolved
– The technical environments have changed but…
– All of these tasks could be done ‘manually’ – at least in principle.
Intelligent Definition and Assurance Slide 13

14. Testers need ThinkingTools
• There are ten testing activities in the New Model
– Test automation tools only support one:‘Applying’
• The remaining nine activities (information
gathering, analysis, modelling, challenging, test
design and so on) are not well supported
• All require some level of thinking and skills
• Checking is possible when a system and its
purpose are well understood and trusted
• Test automation tools are simple in principle…
… compared to the rest of the test process.
Intelligent Definition and Assurance Slide 14

15. Requirements for thinking tools
• The tasks to be supported include:
– Discussing and debating requirements and their sources
– Creating predictive models of system behaviour
– Identifying knowledge gaps; challenging sources
– Creating models of usage, hazards, risks, failure modes,
extreme or erroneous behaviour
– Deciding when a model is adequate or inadequate
– Deciding what to do next from a test outcome
– And so on…
• These are Human or so called Wicked Problems
• For now, tools must focus on the what, not the how.
Intelligent Definition and Assurance Slide 15

16. We can’t solve the Wicked
Problem but…
• “Testing is an information, intelligence or evidence-
gathering activity performed on behalf of (testing)
stakeholders to support their decision-making”
• Can we create tools to support tester
thinking activities and capture that thinking?
• Perhaps the best we can do for now:
– Support human thinking and collaboration
– Look after the paperwork
– Integrate with test automation (the easy part).
Intelligent Definition and Assurance Slide 16

17. Two dimensions of tool capability
• There are several dimensions of tool capability
sophistication perhaps
• Let’s start with a two-dimensional perspective
1. Notetaking, data capture and modelling
capability which I’ll call the ‘Ability to
Capture Knowledge’
2. The second dimension relates more to
knowledge acquisition. Let’s call that the
‘Ability to Investigate’
• I feel a four quadrant model coming on (yes, I
hate them too).
Intelligent Definition and Assurance Slide 17

18. Four quadrant model of intelligent
test tools
Ability to Investigate
AbilitytoCaptureKnowledge
• Text editors, Screen Shots
Models, visualisations, relationships, transformations
• Note Takers
• Mind Maps
• UML/Case Tools
Control,imagination,discernment,foresight
• Pencil and paper, sketching tools
Intelligent Definition and Assurance Slide 18

19. Ability to Capture Knowledge
• Humble text editors and screen shot utilities
• Pencil and paper (better than many software tools)
– Freehand sketches do not limit your imagination
• Dedicated modelling tools using UML are placed highest
– They provide a structure, consistency checking to some degree
and some transformational capabilities which simple drawing or
modelling tools cannot match
– But you are limited to the models the tools can manage
• We may (or may not) have reached half way up this scale
• Tools that give our imagination free reign and perform
validation, consistency checking or transformations, do not
yet exist.
Intelligent Definition and Assurance Slide 19

20. Ability to Investigate
• The lowest capability:
– the tester does all the thinking and has complete
control
• The highest capability:
– the tool is capable of asking its own questions,
discover its own information, make its own models,
judge on the relevance, completeness and accuracy of
the information it acquires
– The tool does all of the thinking required
• Today, all tools are bottom feeders in this respect.
Intelligent Definition and Assurance Slide 20

21. What is this model useful for?
• All of the tools I mention are on extreme left,
mostly towards the bottom left
• Is the model useful for anything?
• It’s less about classification of tools; it’s more a
suggestion of the roadmap our tools might
take
• Let’s consider the situation from another
perspective – that of the medical profession.
Intelligent Definition and Assurance Slide 21

22. Compare the diagnosis of illnesses
to testing
• Doctors ask questions, look for symptoms, take
measurements
• Many ailments can be identified within a few minutes,
most within hours
• Well defined procedures can be performed by bots*
• Doctors won’t be replaced by bots soon because
– Patients like dealing with humans
– Doctors are a powerful lobby (in the UK at least)
• Testers can’t rely on their lobbying power or public
support to resist automation of their roles.
* “The Robot Will See You Now”
http://www.theatlantic.com/magazine/archive/2013/03/the-robot-will-see-you-now/309216/
Intelligent Definition and Assurance Slide 22

23. Future ofTools
What tools can we expect to emerge
in the next few years?

24. Vendors and the tools market
• To date, the tool vendors have picked the low-
hanging fruit of Mechanical Tools
– The market for test automation is crowded
– Open source tools are on the march
• The unexploited market in tools that support
system exploration, collaboration and test design
could be much larger than test execution tools
at least
• All testers need them
– (how many testers? 1 million, 2 million, X million?)
Intelligent Definition and Assurance Slide 24

25. Exploration support
• Frustration with testers:
– testers are unimaginative, working by-rote
– constant pressure to cut costs
• Productivity of exploratory test approaches is proven
• Testers want to explore, but the need for control and
documentation constrains them
• Testers needs tools that can capture plans and tester
activity in real-time
• The next generation will be led by tools that support
the exploration of sources of knowledge.
• These tools might use a “Surveying” metaphor.
Intelligent Definition and Assurance Slide 25

26. A new test process?
• The “tester as surveyor” affects the relationship of
testing to development
• A new style of testing process emerges:
– Test documentation not created in a knowledge vacuum
– Iterative, incremental knowledge acquisition and capture
process closely aligned with the delivery of features
• Could this be an Agile test process at last?
• At least: it fits the increasingly popular Continuous
Delivery, DevOps development approaches.
Intelligent Definition and Assurance Slide 26

27. System Surveying
• A System Survey captures features and the architecture of
the system from a test perspective
– Testers pair with developers and survey features
– The knowledge required to design and build systems emerges
over time
– So do the models produced by testers
• Surveys that evolve the System Model/Map are shared
• The tester surveys paths through the architecture
– Model connections are derived from the paths of exploration
• No need for extensive scripts or test procedures!
– Heard that one before?
– The information required for scripting is in the model.
Intelligent Definition and Assurance Slide 27

28. A scaleable, automatable process
• Test process comprises a sequence of parallel actions
– Sequence: survey, model refinement then testing
– Parallel: small subsets of functionality selected for surveys
– These processes are both iterative and incremental as learning
proceeds
• Scalable: if you survey it, you can test it
• Automatable: What you can survey and test, you can
probably automate
• “Humans make the early maps; tools will follow the trails we
make.”
• We don’t need Machine Learning to do this:
– Simple tools make suggestions that better inform and enrich
exploration and testing.
Intelligent Definition and Assurance Slide 28

29. What effect will Machine Learning
have on testers?
• Tester surveys are the source of data for bots:
– Queries, observations, ideas, concerns mapped to the
system model are a source of data for analysis
– We will need a format and protocol for the information
we capture for the bots to work their magic
• More likely that developers are affected by ML
– In a few years, some component development and unit
testing could be wholly automated
– It would remove a little of the uncertainty that testers face
and may make the tester job a little easier
• We’ll have to wait a bit longer for TerminatorTester.
Intelligent Definition and Assurance Slide 29

30. Intelligent Definition and Assurance Slide 30
TERMINATOR
TESTER
Not Yet!

31. Will Robots ReplaceTesters?
@paul_gerrard
Paul Gerrard
paul@gerrardconsulting.com
gerrardconsulting.com