My talk on Exploratory Testing basics and its future at Belgium Testing Days / BNTQB Test Summit 2017 https://btdconf.org/ bit of slides revamp included

1. BTD and BNTQB Test Summit in Brussels October 25, 2017
Kari Kakkonen, Knowit Oy, FiSTB, ISTQB
@kkakkonen
Exploratory testing –
basics, experiences and
future?
Knowit 1

2. Kari Kakkonen
ROLES
• Knowit Oy, Director/Quality and Competences, Lead Consultant, Trainer and Coach
• Treasurer of ISTQB Executive Committee
• Chairman of Finnish Software Testing Board (FiSTB)
• Auditor of Robot Framework Foundation.
ACHIEVEMENTS
• Influencing testing since 1996
• Ranked in 100 most influential IT persons in Finland (Tietoviikko magazine)
• Great number of presentations in Finnish and international conferences
• TestausOSY/FAST founding member.
• Co-author of Agile Testing Foundations book
• Regular blogger in Tivi-magazine
EDUCATION
• ISTQB Expert Level Test Management Full & Advanced Full & Agile Tester certified
• DASA DevOps Fundamentals, Scrum Master and SAFe certified
• SPICE provisionary assessor certified
• M.Sc.(Eng), Helsinki University of Technology (present Aalto University), Otaniemi, Espoo
• Marketing studies, University of Wisconsin-Madison, the USA.
BUSINESS DOMAINS
• Wide spread of business domain knowledge
• Embedded, Industry, Public,
• Training, Telecom, Commerce,
• Insurance, Banking, Pension
SERVICES
• ISTQB Advanced, Foundation and Agile Testing +
Knowit Quality Professional
• DASA DevOps Fundamentals
• Quality & Test process and organization
development, Metrics
• Agile testing, Scrum, Kanban, Lean
• Leadership
• Test automation, Mobile, Cloud, DevOps
• Quality, Cost, Benefits.
Twitter: @kkakkonen
LinkedIn: fi.linkedin.com/in/karikakkonen/
Knowit 2

4. What is it?
Knowit 4

5. Characterization of exploratory testing
• ”Planning and execution of testing is done at the same time”
(After James Bach)
• Test cases are not necessarily documented even afterwards (Cem Kaner)
• In exploratory testing, test design and test execution occur at the same time, guided by a prepared test
charter. (ISTQB Agile Tester syllabus)
• Testing is done iteratively piece by piece
• Continuous learning and interpretation
• Utilization of knowledge gained from experience
5© Copyright Knowit Oy 2015 | Confidential

6. Exploratory testing - terms
• Test Charter
• Divide your testing areas and assign missions
to each
• Testing area: a bunch of functionalities
• Exploration
• Adventure may go to sidetrack as long as you
come back to mainroad again (Kaner)
• Testing session
• Duration about ½ - 2 hours
• Why?
• Time span of concentrated work is about
20 minutes
• Getting back to work takes about 20
minutes
6© Copyright Knowit Oy 2015 | Confidential

7. Learning in exploratory testing
7© Copyright Knowit Oy 2015 | Confidential
Testing
Opinion-forming
Reporting
Designing actions
Observations
After reference: Psychology of Usability, Sinkkonen et al.

8. Learning from the system and customers
• What has changed or changes frequently?
• What do the customers want?
• What has been defined in an ambiguous way?
• Where do faults cluster?
• Weaknesses in the platform or programming language
• System dependencies
8© Copyright Knowit Oy 2015 | Confidential
Reference: Lessons learned in software testing. Kaner, Bach, Pettichord

9. Learn from other testers, designers, from yourself
• What kind of errors do certain programmers make and how to report to and communicate with them
• What typical errors can there be in the system
• What functionalities have been built in a hurry
• What have you misunderstood and what is typically misunderstood
• How can the system be tested (especially in pair testing!)
9© Copyright Knowit Oy 2015 | Confidential

10. Why exploratory testing?
Knowledge-based perspective
• Exploit the natural diversity of people in testing *)
• ”Do not plan for store”
• Systematic variation of testing
• Quick feedback to designers / developers intensifies learning process *)
• Spread the knowledge of a tester of a special area
10© Copyright Knowit Oy 2015 | Confidential
*) Reference: Exploratory testing: A multiple case study. Itkonen,
Rautiainen

11. Why exploratory testing?
Testers’ approach
• Want to add more test cases and increase the coverage of the tests *)
• For defining the degree of automation of tests
• Quick overview of the quality *)
• Testing the side effects of changes – scripted testing can end up testing only the documented features *)
• The problem in regression testing is the selection of test cases, which requires user experience and
understanding of the system
• When the test documentation can not be written in advance *)
11© Copyright Knowit Oy 2015 | Confidential
*) Reference: Exploratory testing: A multiple case study. Itkonen,
Rautiainen

12. Why exploratory testing?
Management approach
• Low management costs of test documentation
• Finding out the features of a poorly-documented component
• Optimizing the productivity of testing?
• When the available workload is limited *)
• When you want to train the customer support at the same time
12© Copyright Knowit Oy 2015 | Confidential
*) Reference: Exploratory testing: A multiple case study. Itkonen,
Rautiainen

13. Where exploratory testing fits
13
Different styles of testing
Adhoc testing
Exploratory Testing
And Session-Based
Test Management
High level test cases
design and
execution
Detailed test cases
design and execution
ET + SBTM
High level TC
Detailed TC
Automation
Automated testing
and ATDD
Exploratory
Testing
ET
Systematicnessoftesting
Adhoc
ET Scripted testing

14. Exploratory testing as part of test strategy
14© Knowit Oy Confidental 2012
Pure (100%)
scripted testing
Scripted testing
Exploratory testing
Pure (100%)
exploratory testing
Where do you put the
focus of your own
testing?Maybe here?

15. How is it done?
Knowit 15

16. Step by step
© Copyright Knowit Oy 2015 | Confidential
Plan
• Test charter
Test
sessions
• Notes / Log
• Bugs
Debriefings
• Dashboard
”1 hour”
Most of
the time
5 min
regularly

17. Exploratory testing in Prague – find a park
• Test charter
• ”look for green”
1727.10.2017 © Copyright Knowit Oy 2013 | Confidential | Version 1.0

18. Exploratory testing in Prague – find a park
• Test execution log
• Dashboard
1827.10.2017 © Copyright Knowit Oy 2013 | Confidential | Version 1.0

19. Exploratory testing in Prague – find a park
• Defect report
1927.10.2017 © Copyright Knowit Oy 2013 | Confidential | Version 1.0

20. Purpose of test charter(s)
• What will be tested?
• What documents are available?
• What kind of errors are being sought?
• Tasks and what test techniques will be used?
• What will be expected from test sessions?
• A dashboard?
• Defect reports?
• Some key test cases?
20© Copyright Knowit Oy 2015 | Confidential
Reference: A practioner’s guide to software test design. Copeland
Ref. Exploratory testing: A multiple case study. Itkonen, Rautiainen

21. Test design within test charter
• Define the testing areas of the test object
• E.g. 20-40 areas covering whole system under test
• Divide each area to one or more test sessions
• Test charter works as a roadmap for each test session
• Define test cases to be documented as a by-product, if any
• heuristic: less than 10% of all tests
• Write down test ideas and/or high level test cases
21© Copyright Knowit Oy 2015 | Confidential

22. Charter classical example
Area Coverage
and
working
hours
Practice Documents Result possible
errors
Risks
R1. Customer’s
all selected
items are not
added to order,
Effect: 20
eur/buyer,
probability 5%
R2. Order can
not be
completed after
interruption, 5,
probability: ??
Main page 100%
Path coverage
(direct paths) and
the most common
(80% used) loops
10h
Scripting
with
Functional
Tester-tool
Main page display
description
document, navigation
map (COMING
FROM
DEVELOPMENT)
All pages and
the shopping
cart are
available
Shopping
cart
5h Shopping cart-
UC.doc (use case)
Shopping cart
can be used in
the same way
as a real
shopping cart
The same
product can
not be added
to shopping
cart several
times
Emptying the
shopping cart
causes an
execption
R1
Order ? Order-UC.doc? R2
22© Copyright Knowit Oy 2015 | Confidential
Reference: James Bach

23. Test Charter example ISTQB
23© Copyright Knowit Oy 2017 | Confidential
• Actor: intended user of the system
• Purpose: the theme of the charter including what particular objective the
actor wants to achieve, i.e., the test conditions
• Setup: what needs to be in place in order to start the test execution
• Priority: relative importance of this charter, based on the priority of the
associated user story or the risk level
• Reference: specifications (e.g., user story), risks, or other information
sources
• Data: whatever data is needed to carry out the charter
• Activities: a list of ideas of what the actor may want to do with the system
and what would be interesting to test (both positive and negative tests)
• Oracle notes: how to evaluate the product to determine correct results
• Variations: alternative actions and evaluations to complement the ideas
described under activities
Reference: ISTQB Agile Tester syllabus

24. Charter as an excel
24© Copyright Knowit Oy 2015 | Confidential

25. Test charter and test log in mindmap (Xmind)
25© Copyright Knowit Oy 2015 | Confidential

26. Documents supporting testing
• Charter
• List of different testing strategies
• Lists of heuristics
• List of typical errors
• Kaner’s bug taxonomies*)
• Legal notices, standards, de facto-standards
• Requirements and design documentation of the system
• Self made description and models of the system behavior
• User guide *)
26© Copyright Knowit Oy 2015 | Confidential
*) Reference: Testing Computer Software. Kaner et al.
**) Reference: Exploratory testing: A multiple case study. Itkonen, Rautiainen

27. Kaner’s bug taxonomies
• User interface errors
• Error handling
• Boundary-related errors
• Calculation errors
• Initial and later states
• Control Flow errors
• Errors in handling or interpreting data
• Race conditions
• Load Conditions
• Hardware
• Source and version control
• Documentation
• Testing errors
27© Copyright Knowit Oy 2015 | Confidential
Reference: Testing computer software, p. 60 – 64

28. Exploratory Testing - Heuristics
• A set of heuristics can be applied when testing.
• A heuristic can guide the tester in how to perform the testing and to evaluate the results, few examples
below
28© Copyright Knowit Oy 2015 | Confidential
Heuristics Examples
Boundaries: Approaching the Boundary e.g. almost too big, almost too
small, at the Boundary
CRUD Create, Read, Update, Delete
Configuration variations Varying the variables related to configuration e.g. Screen
Resolution; Network Speed, Latency, Signal Strength; etc.
Interruptions e.g., log off, shut down, or reboot
Source: Elisabeth Hendrickson www.testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf

29. Performance attitude - professional working
• Keep the mission, your targets of testing in mind
• Write down observations and questions about the system
• Report in a disciplined and systematical way
• During the execution, write only the most essential test cases and in high level
• Test cases can be refined later, if needed
29© Copyright Knowit Oy 2015 | Confidential

30. Tests are designed during execution
• Design test on the basis of charter its initial test
ideas
• Define tests from any questions in your mind
• A surprising situation may indicate an error:
Utilize the surprise effect!
• Use heuristics e.g.
• ”Backwards thinking”: ”This button saves
the definition text. I wonder what other ways
are there for saving the text?”
30© Copyright Knowit Oy 2015 | Confidential

31. Note taking: test execution logs
• Keep a test execution log / notes
• Keep track of the tests carried out
• Main thing is that executed tests are noted down, not the detail
• You may create scripted test cases from some of the tests, add detail
• Note what test data you used in your tests
• You may record your execution with a suitable tool, only need to add your comments
• Add status to your notes e.g. with color-coding
• Write down also notes for test session post-mortem and your own learning
31© Copyright Knowit Oy 2015 | Confidential

32. Note taking: defect logs
• After session work out defect reports for defects you found, based on your notes
• In defect reporting, traceability to the requirements must be maintained so that coverage can be
evaluated
• A well written execution log is the best evidence of the existence of a fault
• Report a bug clearly, so that the failure can be repeated
• You may use defect reporting systems
32© Copyright Knowit Oy 2015 | Confidential

33. Testing dashboard as a test report - an example
33© Copyright Knowit Oy 2015 | Confidential
Test area Workload Coverage Quality
level/risks
Comment
Main page !Interrupted High,
5h
Very high [all
parts + stress
tests etc..]
49: 1435, 36:
1469,
42: 1501
wait for more
pictures of
user interface
Shopping
cart
!Started
High, 2h (reserved
4h)
Low [main parts
to testing] (High)
81: 1425
[probability 9 x
effect 9; error
number. 1425]
Order !Done
6h (reserved 4h)
High [all parts]
Feedback !Not done
Low (reserved 1h)
Low [main parts
to testing]
Reference: James Bach

34. Test charter and test log in mindmap (Xmind)
34© Copyright Knowit Oy 2015 | Confidential

35. Measuring exploratory testing
• The duration of the session
• The relative change in the number of test cases by the same tester
• Coverage of testing per session
• The number of interruptions (Suspension criteria)
• Number of rejected defects in defect database
• …
• Metrics are the eyeglasses of testing that you need in order to be fully aware of the situation and
potential problems in testing
•  It is recommended that you choose metrics that are suitable for the challenges of exploratory
testing
35© Copyright Knowit Oy 2015 | Confidential

36. Some Tools
• Notetaking:
• Screenshots With Annotation
• Video with Annotation
• Intergrated bug reporting
• Test Management tool suite add-ons
• Micro Focus / HPE Sprinter
• QA Symphony qTest eXplorer
• Telerik Test Studio Explore
• qMetry Voyager
• Steps Recorder
• Recording Browser extensions
• Bug Magnet
• Notepad++ (pad tools in general)
• Xmind, Mindmup (mindmap tools in general)
• Rapid Reporter
36© Copyright Knowit Oy 2015 | Confidential
• Test charter planning
• Charters
• Priorization
• Test Management tool suites
• Trello
• Jira Agile
• Micro Focus / HPE Agile Manager
• Pivotal Tracker
• Excel

37. Who can do it?
Knowit 37

38. Who to recruit? The profile of an exploratory tester
• Exploratory testing is particularly well suitable for a person, who…
• likes to take risks
• is not afraid of changes or new things
• is open-minded
• sets challenges and goals to themselves
• is smart and quick in finding test conditions
• ”Pioneer-style”
• Everyone can learn to be an exploratory tester
38© Copyright Knowit Oy 2015 | Confidential
Reference: Choosing and Managing the Ideal Test Team. Lloyd Roden

39. Exploratory testing requires learning skills
• Outline the functionality of the system on paper
• Aim at understanding
• Don’t force yourself to remember facts, use documents
• Ask questions about the functionality of the system
• Recognize the items on which you need more information
39© Copyright Knowit Oy 2015 | Confidential
Reference: Tutkiva oppiminen. Hakkarainen et al.

40. Improve your testing thinking skills
• Observe your own testing habits
• Recognize your own ways of thinking
• Learn from misunderstandings and mistakes
• Select the testing techniques according to the situation *)
• Improve your skills to deduce the states of the system *)
43© Copyright Knowit Oy 2015 | Confidential
Adapted from the reference: Tutkiva oppiminen. Hakkarainen et al.,
*) Reference: Rapid Software Testing. James Bach

41. What in future?
Knowit 44

42. Exploratory future
• Exploratory testing will become the norm in agile testing
• And agile will be the norm for all development
• Testing tools suites will include exploratory testing charters and logging
• Coverage will be proven with both last quantities of well-logged exploratory testing and some formally
designed test cases
• With test automation to make sure the basics work, testers will be free to find important defects with
exploratory testing
• People will start exploring also with test automation
• Keyword driven and data-driven approaches
• AI will help with some of exploratory Testing
© Copyright Knowit Oy 2015 | Confidential

43. Questions?
kari.kakkonen@knowit.fi
@kkakkonen

44. 10/27/2017 Knowit 47
Creating digital
opportunities