Some basics of exploratory testing coupled with experiences, hints and tools for exploratory testing, and wrapped up by a view to the future of exploratory testing. Presentation given 19.10.2016 in Soeul, South Korea in SSTC and TMMi international conference 2016.

1. Exploratory testing –
basics, experiences and future
Kari Kakkonen/ kari.kakkonen@istqb.org

2. 1. What is Exploratory Testing
2. How is Exploratory Testing done?
3. Who can do it?
4. What in future?

3. Kari Kakkonen, Knowit
• Speaks, train, coach and mentor regularly
about
• ISTQB Advanced, Foundation and Agile
Testing + Knowit Quality Professional
• Quality & Test process and organization
development
• Agile testing, Scrum, Kanban, Lean
• Metrics
• Leadership
• Test automation, mobile, cloud, DevOps
• Quality, Cost, Benefits
• Speaking & writing highlights
• EuroSTAR and Iqnite several times
• ASTQB in USA, OOP in Germany, TEST-IT in
South-Africa, Nordic Testing Days in Estonia,
Testing Days in Czech, Israel Testing Week
• Numerous times in Finland at Testing
Assembly, Aalto Testing Days, Tieturi Testing,
Talentum Testing Forum, Quality Assurance &
Software Testing, ICT Expo, TestIT Summit,
Microsoft, HP, IBM, Borland etc. events
• Testing Experience magazine, Quality and
Testing magazine, Sytyke-magazine,
Tietoviikko
• Education
• ISTQB Expert Level Test Management Full &
Advanced Full & Agile Tester certified
• SAFe and Scrum Master certified
• SPICE provisionary assessor certified
• M.Sc, Helsinki University of Technology / Aalto-
university
• Marketing studies, University of Wisconsin-
Madison
• Professional achievements
• Wide spread of business domain knowledge
• Embedded, Industry, Public,
• Training, Telecom, Commerce,
• Insurance, Banking, Pension
• ISTQB Treasurer, Executive Committee 2015-
• Finnish Software Testing Board FiSTB, chairman
• TestausOSY/FAST founding member
• Knowit, Director, Quality and Competences
• Chairman of research project STX, Lappeenranta
University of Technology
• Finnish Software Measurement Association FiSMA
ry ex-board member
• Ranked in 100 most influential IT-persons in
Finland
© Knowit Oy
Twitter: @kkakkonen
LinkedIn: fi.linkedin.com/in/
karikakkonen/
kari.kakkonen@knowit.fi

4. 100+
Mobile
apps
20+
Extranet
services
50+
Intranet
services
25+
Web stores
500+
Web sites100+
Service design
projects
Knowit – We are
known for our
work
We develop and grow
our customers’
business.
© Copyright Knowit Oy 2015 | Confidential | Version 1.0
#1
In Quality
Assurance

5. What it is?
© Copyright Knowit Oy 2015 | Confidential 5

6. CHARACTERIZATION OF EXPLORATORY
TESTING
• ”Planning and execution of testing is done at the same time”
(After James Bach)
• Test cases are not necessarily documented even afterwards (Cem
Kaner)
• Testing is done iteratively piece by piece
• Continuous learning and interpretation of conceptions
• Utilization of knowledge gained from experience
© Copyright Knowit Oy 2015 | Confidential 6

7. EXPLORATORY TESTING - TERMS
• Adventure may go to sidetrack as
long as you come back to
mainroad again (Kaner)
• Testing area: a bunch of
functionalities
• Testing session
• Duration about ½ - 2 hours
• Time span of concentrated work is
about 20 minutes
• Getting back to work takes about
20 minutes
© Copyright Knowit Oy 2015 | Confidential 7

8. AN EXAMPLE: WWW.TOPSELLERS.COM, A
FICTIONAL E-COMMERCE SHOP
• Test case: ”Log in. Browse the content of your shopping cart. Result:
Shopping cart is empty, because no items have been picked.”
• During testing it is noticed that the shopping cart contains items! Items
have been picked with the same user account and the e-commerce
shop keeps the earlier picked things in shopping cart.
• Aftertaste: ”The test should have considered this. Let’s change the test
data and the test itself, and design a new test case, which takes into
account that the shopping cart can store items.”
• A familiar situation?
© Copyright Knowit Oy 2015 | Confidential 8

9. LEARNING IN EXPLORATORY TESTING
Testing
Opinion-forming
Reporting
Designing actions
Observations
After reference: Psychology of Usability, Sinkkonen et al.
© Copyright Knowit Oy 2015 | Confidential 9

10. How is it done?
© Copyright Knowit Oy 2015 | Confidential 15

11. Step by step
© Copyright Knowit Oy 2015 | Confidential
Plan
• Test charter
Test
session
• Notes / Log
• Bugs
Debriefing
• Dashboard

12. Exploratory testing in Prague – find a park
• Test
charter
• ”look for
green”
19.10.2016 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 17

13. Exploratory testing in Prague – find a park
• Test execution log
• Dashboard
19.10.2016 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 18

14. Exploratory testing in Prague – find a park
• Defect report
19.10.2016 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 19

15. TEST DESIGN
• Define the testing areas of the test object
• Divide each area to one or more test sessions
• Test charter works as a roadmap per test session
• Define test cases to be documented
• heuristic: less than 10% of all tests
• Write down test ideas and/or high level test cases
© Copyright Knowit Oy 2015 | Confidential 20

16. PURPOSE OF TEST CHARTER(S)
• What will be tested?
• What documents are available?
• What kind of errors are being sought?
• Tasks and what test techniques will be used?
• Targets and outputs (for example reports)
Reference: A practioner’s guide to software test design. Copeland
Ref. Exploratory testing: A multiple case study. Itkonen, Rautiainen
© Copyright Knowit Oy 2015 | Confidential 21

17. Charter
© Copyright Knowit Oy 2015 | Confidential
22
Area Coverage
and
working
hours
Practice Documents Result possible
errors
Risks
R1. Customer’s
all selected
items are not
added to order,
Effect: 20
eur/buyer,
probability 5%
R2. Order can
not be
completed after
interruption, 5,
probability: ??
Main page 100%
Path coverage
(direct paths) and
the most common
(80% used) loops
10h
Scripting
with
Functional
Tester-tool
Main page display
description
document, navigation
map (COMING
FROM
DEVELOPMENT)
All pages and
the shopping
cart are
available
Shopping
cart
5h Shopping cart-
UC.doc (use case)
Shopping cart
can be used in
the same way
as a real
shopping cart
The same
product can
not be added
to shopping
cart several
times
Emptying the
shopping cart
causes an
execption
R1
Order ? Order-UC.doc? R2

18. Charter as an excel
© Copyright Knowit Oy 2015 | Confidential 23

19. DOCUMENTS SUPPORTING TESTING
• Charter
• List of different testing strategies
• Lists of heuristics
• List of typical errors
• Kaner’s bug taxonomies*)
• Legal notices, standards, de facto-standards
• Requirements and design documentation of the system
• Self made description and models of the system behavior
• User guide *)
*) Reference: Testing Computer Software. Kaner et al.
© Copyright Knowit Oy 2015 | Confidential 24
**) Reference: Exploratory testing: A multiple case study. Itkonen, Rautiainen

20. Kaner’s bug taxonomies:
(Testing computer software, p. 60 – 64)
• User interface errors
• Error handling
• Boundary-related errors
• Calculation errors
• Initial and later states
• Control Flow errors
• Errors in handling or interpreting data
• Race conditions
• Load Conditions
• Hardware
• Source and version control
• Documentation
• Testing errors
© Copyright Knowit Oy 2015 | Confidential 25

21. Exploratory Testing - Heuristics
• A set of heuristics can be applied when testing.
• A heuristic can guide the tester in how to perform the testing and to
evaluate the results, few examples below
© Copyright Knowit Oy 2015 | Confidential 26
Heuristics Examples
Boundaries: Approaching the Boundary e.g. almost too big,
almost too small, at the Boundary
CRUD Create, Read, Update, Delete
Configuration variations Varying the variables related to configuration e.g.
Screen Resolution; Network Speed, Latency, Signal
Strength; etc.
Interruptions e.g., log off, shut down, or reboot
Source: Elisabeth Hendrickson www.testobsessed.com/wp-content/uploads/2011/04/testheuristicscheatsheetv1.pdf

22. PERFORMANCE ATTITUDE -
PROFESSIONAL WORKING
• Keep the targets of testing in mind
• You can visit bypaths but only for a moment
• Write down observations and questions about the system
• Report in a disciplined and systematical way
• During the execution, write only the most essential test cases and in
high level
• Test cases can be refined later
© Copyright Knowit Oy 2015 | Confidential 27

23. TESTS ARE DESIGNED DURING
EXECUTION
• Define a test from a question
• Design test on the basis of charter and test
ideas
• A surprising situation may indicate an error:
Utilize the surprise effect!
• Heuristics e.g.
• ”Backwards thinking”: ”This button saves the
definition text. I wonder what other ways are
there for saving the text?”
© Copyright Knowit Oy 2015 | Confidential 28

24. NOTE TAKING: TEST EXECUTION LOGS
• Keep a test execution log
• Keep track of the tests carried out
• Main thing is that executed tests are noted
• You may create scripted test cases of some of the tests
• Keep the most important test cases that have been executed, which show
how the testing has been done, e.g. what values have been used
• You may record your execution
• Write down also test notes for test session post-mortem and your own
learning
© Copyright Knowit Oy 2015 | Confidential 29

25. Test charter and test log in mindmap (Xmind)
© Copyright Knowit Oy 2015 | Confidential 30

26. NOTE TAKING: DEFECT LOGS
• In defect reporting, traceability to the requirements must be
maintained so that coverage can be evaluated
• A well written error log is the best evidence of the existence of a fault
• Report a bug clearly, so that the failure can be repeated
• You may use defect reporting systems
© Copyright Knowit Oy 2015 | Confidential 31

27. TESTING DASHBOARD AS A TEST REPORT
- AN EXAMPLE
© Copyright Knowit Oy 2015 | Confidential 32
Test area Workload Coverage Quality
level/risks
Comment
Main page !Interrupted High,
5h
Very high [all
parts + stress
tests etc..]
49: 1435, 36:
1469,
42: 1501
wait for more
pictures of
user interface
Shopping
cart
!Started
High, 2h (reserved
4h)
Low [main parts
to testing] (High)
81: 1425
[probability 9 x
effect 9; error
number. 1425]
Order !Done
6h (reserved 4h)
High [all parts]
Feedback !Not done
Low (reserved 1h)
Low [main parts
to testing]

28. MEASURING EXPLORATORY TESTING
• The duration of the session
• The relative change in the number of test cases by the same tester
• Coverage of testing per session
• The number of interruptions (Suspension criteria)
• Number of rejected defects in defect database
• …
• Metrics are the eyeglasses of testing that you need in order to be fully
aware of the situation and potential problems in testing
 It is recommended that you choose metrics that are suitable for the
challenges of exploratory testing
© Copyright Knowit Oy 2015 | Confidential 33

29. Some Tools
• Notetaking:
• Screenshots With Annotation
• Video with Annotation
• Intergrated bug reporting
• HP Sprinter
• QA Symphony qTest eXplorer
• Telerik Test Studio Explore
• Bug Magnet
• Notepad++ (pad tools in general)
• Xmind (mindmap tools in general)
• Rapid Reporter
• Pivotal Tracker
• Test charter planning
• Task and backlog tools
• Test charter planning
• Trello
• Jira Agile
• HP Agile
• Excel
© Copyright Knowit Oy 2015 | Confidential 34

30. Who can do it?
© Copyright Knowit Oy 2015 | Confidential 35

31. WHO TO RECRUIT? THE PROFILE OF AN
EXPLORATORY TESTER
• Exploratory testing is particularly well suitable for a person, who…
• likes to take risks
• is not afraid of changes or new things
• is open-minded
• sets challenges and goals to themselves
• is smart and quick in finding test conditions
• ”Pioneer-style”
• Everyone can learn to be an exploratory tester
Reference: Choosing and Managing the Ideal Test Team. Lloyd Roden
© Copyright Knowit Oy 2015 | Confidential 36

32. EXPLORATORY TESTING REQUIRES
LEARNING SKILLS
• Outline the functionality of the system on paper
• Aim at understanding
• Don’t force yourself to remember facts, use documents
• Ask questions about the functionality of the system
• Recognize the items on which you need more information
Reference: Tutkiva oppiminen. Hakkarainen et al.
© Copyright Knowit Oy 2015 | Confidential 37

33. IMPROVE YOUR TESTING THINKING
SKILLS
• Observe your own testing habits
• Recognize your own ways of thinking
• Learn from misunderstandings and mistakes
• Select the testing techniques according to the situation *)
• Improve your skills to deduce the states of the system *)
Adapted from the reference: Tutkiva oppiminen. Hakkarainen et al.,
*) Reference: Rapid Software Testing. James Bach
© Copyright Knowit Oy 2015 | Confidential 41

34. What in future?
© Copyright Knowit Oy 2015 | Confidential 42

35. Exploratory future
• Exploratory testing will become the norm in agile testing
• And agile will be the norm for all development
• Coverage will be proven with both last quantities of well-logged
exploratory testing and some formally designed test cases
• With test automation to make sure the basics work, testers will be free
to find important defects with exploratory testing
• People will start exploring also with test automation
• Keyword driven and data-driven approaches
© Copyright Knowit Oy 2015 | Confidential

36. Thank you
@kkakkonen
kari.kakkonen@istqb.org