Risk Mitigation Using Exploratory and Technical Testing | QASymphony Webinar
1. Risk Mitigation Using
Exploratory andTechnical
Testing
28th June 2016
Alan Richardson â Compendium Developments Ltd
Join the conversation â use the hashtag #risktesting on Twitter
The audio for this webinar is delivered through your computer. There is no dial-in
number. Make sure your speakers are turned up or use a pair of headphones.
2. HOWTOWATCHTHISWEBINAR
Join the conversation â use the hashtag #risktesting on Twitter
⢠Audio for this webinar is delivered through your computer. Make
sure your speakers are turned up or use a set of headphones.
⢠If your audio quality is choppy, it could be your internet connection.
⢠You can customize your webinar viewing experience by
increasing, decreasing or minimizing the size of the widgets on
your screen.
⢠If you have questions, enter them in the widget on the left.
3. ⢠This webinar will be recorded and available on-demand tomorrow.
You will get an email when it is available.
⢠Join the conversation on Twitter using the hashtag #risktesting
⢠Use the Q&A widgets to ask questions during the webinar.
⢠At the end of the webinar, you will be asked to take a short survey.
HOUSEKEEPING
Join the conversation â use the hashtag #risktesting on Twitter
4. Robust test management platform purpose-built to help agile teams
centralize, organize and accelerate software testing
ABOUT QASYMPHONY
5. OTHER WEBINARS & RESOURCES
Join the conversation â use the hashtag #risktesting on Twitter
WWW.QASYMPHONY.COM/RESOURCES
6. Guest Speaker: Alan Richardson
⢠Alan has worked in Software Development for
over 20 years; as a programmer, tester, test
manager. As an independent test consultant he
helps organisations improve their agility,
technical skills and testing processes.Alan
wrote the books "Dear EvilTester", "Java For
Testers" and "Selenium Simplified"; he also
created online training courses on technical
web testing, Java and SeleniumWebDriver.
⢠Alan blogs at EvilTester.com,
SeleniumSimplified.com, and
JavaForTesters.com; you can find information
on his consultancy, training and conference
talks at CompendiumDev.co.uk. Follow him on
twitter as @EvilTester.
OUR PRESENTER
Speaker
Headshot
BRANDING OR
PROMOTION
CompendiumDev.co.uk
Join the conversation â use the hashtag #risktesting on Twitter
7. Everyone is already familiar with riskâŚ
⢠Differences for thisWebinar
âRisk as a belief model
âRisk underpins testing, so use risk
Âťto derive and change process
Âťto explore more
Âťto push testing further
Âťto become more technical
Join the conversation â use the hashtag #risktesting on Twitter
8. Commonsense Risk
⢠What is risk?
âSomething that might go wrong
⢠Probability
⢠Impact
Join the conversation â use the hashtag #risktesting on Twitter
9. Commonsense Webinar Risks
⢠What might go wrong
â With me
⢠What if Iâm ill?What if I still
have a cold and canât talk?
⢠What if I forget what Iâm
talking about?
â With my broadband
⢠What if the connection
drops?What if the speed is
poor?
â With my computer
⢠What if it crashes?
â With the webinar system
⢠What if it stops?
⢠What might go wrong
â With the phone?
⢠What if it cuts out?
â With the locale
⢠What if there is a power cut?
â With the content
⢠What if I bore people?
⢠What if they drop out?
Join the conversation â use the hashtag #risktesting on Twitter
10. Commonsense Risk Process
⢠Identify
⢠Mitigate
⢠Detect
⢠Accept
Join the conversation â use the hashtag #risktesting on Twitter
11. So with the WebinarâŚ
⢠Identify
⢠Mitigate
â Illness â sleep more, pre-record webinar just in case
â Forget â presenter notes, practice
â Broadband â give slides to host
â Computer crash â multiple computers
â Power cut â battery, UPS
â Phone â landline, mobile
⢠Accept
â Boredom, Drop out
⢠Detect
â Have computer watching webinar â risk: impacts Mbps
Join the conversation â use the hashtag #risktesting on Twitter
12. Risk IsâŚ
⢠Everywhere
⢠Associated with EveryThing
⢠Inherent in Every Process
⢠All pervasive
https://www.flickr.com/photos/britishlibrary/11065829793
Join the conversation â use the hashtag #risktesting on Twitter
13. Risk Example: Contact Form on Web Site
You received this e-mail message through your website:
reason: default
E-mail: ngjchr@somewebsitethatdoesnotexist.com
Name: ewogwah
Message: pK9ctN kfoummnkudob,
[url=http://dnaaimbzpgyg.com/]dnaaimbzpgyg[/url],
[link=http://mmkwfndaydxb.com/]mmkwfndaydxb[/link],
http://bwtjxnpecomy.com/
:
IP: 46.161.9.32
Browser: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Points: 0
Join the conversation â use the hashtag #risktesting on Twitter
14. Opportunity: Contact Form doubles as Web
Site is âupâ checker
You received this e-mail message through your website:
reason: default
E-mail: ngjchr@somewebsitethatdoesnotexist.com
Name: ewogwah
Message: pK9ctN kfoummnkudob,
[url=http://dnaaimbzpgyg.com/]dnaaimbzpgyg[/url],
[link=http://mmkwfndaydxb.com/]mmkwfndaydxb[/link],
http://bwtjxnpecomy.com/
:
IP: 46.161.9.32
Browser: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Points: 0
Web Site Status
Join the conversation â use the hashtag #risktesting on Twitter
15. Risk & Opportunity
⢠What is risk?
â Something that might go wrong
⢠Probability
⢠Impact
Opportunity for
testing
Join the conversation â use the hashtag #risktesting on Twitter
16. General Risks Relating to Testing
⢠The functionality might not work
â âFunctional Condition Riskâ
⢠There is a risk that this change might have a knock on
effect in the system
â âRegression Riskâ, âChange Riskâ
Create
Process to
mitigate risk
Join the conversation â use the hashtag #risktesting on Twitter
17. You probably already use risk in your testing
â˘Business Risk
â˘Project Risk
â˘Functional Risk
Join the conversation â use the hashtag #risktesting on Twitter
18. Typical Risk Modeling: Business Risks
⢠We might run out of funding
⢠Our requirements might be wrong
⢠We might be hit by a regulatory requirement
Less likely to be used for testing
Join the conversation â use the hashtag #risktesting on Twitter
19. We usually mean project and functional risk
⢠Project Risk
â We might not have enough staff
â Our staff might go sick
â Everyone takes holiday at the same time
â Our business users donât know the requirements
â Our business users change their requirements
â etc.
⢠Functional risk
â Userâs canât register on the site
â The payment integration fails
â The regulatory reporting fails
â etc.
Test Manager
Test Practitioner
Join the conversation â use the hashtag #risktesting on Twitter
20. Risk
⢠What is risk?
âSomething that might go wrong
⢠Probability
⢠Impact People get hung
up here
Join the conversation â use the hashtag #risktesting on Twitter
21. Priority and Probability Procrastination for
Project Head Person Protection
http://www.slideshare.net/profmcgill/risk-
analysis-for-dummies
Mitigate against
Beliefs and fears
about being
wrong
Join the conversation â use the hashtag #risktesting on Twitter
22. Risk & Testing
Join the conversation â use the hashtag #TBD on Twitter
How
does risk
relate to
testing?
23. ISTQB âRisk-Based Testingâ
âAn approach to testing to reduce the level of
product risks and inform stakeholders of their
status, starting in the initial stages of a project.
It involves the identification of product risks
and the use of risk levels to guide the test
process.â
http://www.astqb.org/glossary/search/risk-based%20testing
Join the conversation â use the hashtag #risktesting on Twitter
24. ISTQB Risk Based Testing
âAn approach to testing to reduce the level of
product risks and inform stakeholders of their
status, starting in the initial stages of a project.
It involves the identification of product risks
and the use of risk levels to guide the test
process.â
http://www.astqb.org/glossary/search/risk-based%20testing
Join the conversation â use the hashtag #risktesting on Twitter
25. ISTQB Risk Based Testing
http://www.astqb.org/glossary/search/risk-based%20testing
Mitigation
Detection
Analysis
Prioritization
Derivation
Join the conversation â use the hashtag #risktesting on Twitter
26. Risk Management & Testing
⢠Mitigation
â Do something to make the risk less likely
⢠Detection
â Find out if the risk has manifested as an issue
⢠Analysis
â Identify Risks
⢠Prioritization
â Decide which risks are more important: how bad
is the impact, who is it bad for, how likely to we
believe the risk to be?
⢠Derivation
â How can you test for this: make it manifest,
check if it manifests, explore impact?
Risk
Management
Testing
Join the conversation â use the hashtag #risktesting on Twitter
27. Some Risk Classifications
⢠Functional Risk
â Relating to the functionality
⢠function, security, performance, accessibility
⢠System Risk
â performance, security, backups, install, restore
⢠Technical Risk
â Technology involved: load balancing, libraries,
protocols, platform compatibility
⢠Non-system Related
â Process Risk
â Business Risk
â Project Risk
Based on the System
Of Development
Join the conversation â use the hashtag #risktesting on Twitter
28. Some Risk Classifications
⢠Functional Risk
â Relating to the functionality
⢠function, security, performance, accessibility
⢠System
â performance, security, backups, install, restore
⢠Technical Risk
â Technology involved: load balancing, libraries, protocols,
platform compatibility
⢠Non-system Related
â Process Risk
â Business Risk
â Project Risk
See Related
References
See Related
References
Join the conversation â use the hashtag #risktesting on Twitter
29. Process Risk
⢠System âofâ Development
â How we develop software
â What is our process?
â What are our skills?
â What tools do we use?
â etc.
The way we develop
software opens us up to
different types of risk.
Join the conversation â use the hashtag #risktesting on Twitter
30. Process Risk
⢠System âofâ Development
â How we develop software
â What is our process?
â What are our skills?
â What tools do we use?
â etc.
The way we develop
software opens us up to
different types of risk.
And that is why we
adopt different
approaches in how
we test.
Join the conversation â use the hashtag #risktesting on Twitter
31. Process Risk
Analyse Process
Identify Risks
Any issues that happen?
What works what doesnât?
Change Process ToâŚ
Mitigate Detect
Accept Risk
Join the conversation â use the hashtag #risktesting on Twitter
32. Every Process Has Inherent Risk
⢠âWe use a very structured and
traditional approach to testingâ
Risks:
⢠We canât estimate accurately in
advance
⢠Development over-runs
⢠Testing takes too long at the end
(to meet our schedule)
⢠Testing canât respond fast enough
when requirements change
⢠Test Cases
⢠Test Scripts
⢠Test Plans
⢠Test Strategies
⢠etc.
⢠âWe use waterfall developmentâ
&
Join the conversation â use the hashtag #risktesting on Twitter
33. Process & Culture Clash Risk
⢠âWe use a very structured and
traditional approach to testingâ
Risks:
⢠Testing is too slow
⢠Testing doesnât add value
⢠We donât need testing
⢠Test Cases
⢠Test Scripts
⢠Test Plans
⢠Test Strategies
⢠etc.
⢠âWe use an Agile approach to
developmentâ
&
Join the conversation â use the hashtag #risktesting on Twitter
34. Process as a System
Stories
Conversations Code
Explore
Done
Automate
Time-boxed
Join the conversation â use the hashtag #risktesting on Twitter
36. Risk Driven Process
Stories
Conversations Code
Explore
Done
Automate
Time-boxed
Miscommunication
Misunderstanding,
Omissions, Bugs
Overcommit,
Emergencies
Too much to
automate, wrong
tools
Tech
Debt
Too
early
Discuss âtest ideasâ,
add ideas to story
Log testing done,
debrief, small
chunks, prioritise
iteratively
Early questions, but
not too early
Join the conversation â use the hashtag #risktesting on Twitter
37. Process Risks Are System Risks
⢠InterconnectedTeams
⢠Individuals
⢠Relationships
â Communication
â Artefact delivery and review
⢠Timings
⢠Expectations, Input/Output,
Contracts
⢠Etc.
âSystem of
Developmentâ
Create
Process to
mitigate risk
Join the conversation â use the hashtag #risktesting on Twitter
38. Changing Process is a Risk
But if we already know it doesnât work
how can we justify not changing?
Join the conversation â use the hashtag #risktesting on Twitter
40. Secondary Gain
⢠Unrecognised âbenefitâ
⢠e.g. Smoking
⢠-> Main Risk -> I might die
⢠Secondary Gain
â I get to take breaks
outside
â I get to chat and socialise
â I have stress relief breaks
⢠Secondary Gain means I
might not stop smoking,
even if I try.
https://www.flickr.com/photos/britishlibrary/11103578275/
Join the conversation â use the hashtag #risktesting on Twitter
41. Hypothetical Examples of Secondary Gain
⢠Risk keeps us in business
⢠Process risk justifies our âstandardâ
⢠Not enough time means we never have to finish
⢠Not enough time means we donât have to learn
⢠Secondary Gain is a massive risk to change
â Identify secondary gain
â and change your attitude to it
Join the conversation â use the hashtag #risktesting on Twitter
42. Testing must not be limited by our beliefs
⢠What do I think could go wrong?
âOptions are limited by our model of the world
â5Whys questioning specifically targets beliefs.
âWhat Else?
âSystems Analysis
Join the conversation â use the hashtag #risktesting on Twitter
43. How does âriskâ lead to exploratory testing
⢠I believe
â The more complicated a system the more
risk that something can go wrong
â We want to simplify the âprocessâ as much
as we can
Join the conversation â use the hashtag #risktesting on Twitter
44. How does âriskâ lead to exploratory testing
If you had no test process and designed one based on risk:
I donât know how
to test it What is it
supposed to do?
Who is going
to use it?
What data
does this
process?
Join the conversation â use the hashtag #risktesting on Twitter
45. How does âriskâ lead to exploratory testing
If you had no test process and
designed one based on risk:
I donât know how
to test it What is it
supposed to do?
Who is going
to use it?
What data
does this
process?
Risk: We donât
know if it
works.
Risk: It might
not function.
Risk: It might
not meet user
need.
Risk: It might
not handle
input
Join the conversation â use the hashtag #risktesting on Twitter
46. How does âriskâ lead to exploratory testing
⢠Then we would improve the process by looking at other
risks:
â Risk that we havenât tested enough
⢠Agree high level conditions, review the conditions before we
start, review the work
â Risk that we canât tell people what we did
⢠Learn to take notes, communicate what we do, collate reports in
a searchable form
â Risk that we canât plan it because we donât know what weâll
test
⢠agree a time constraint, work in small chunks, prioritise
coverage, adjust based on review of the output
â etc.
Join the conversation â use the hashtag #risktesting on Twitter
47. Basic System
Login Web Page <-> HTTP Server <-> DB with user details
HTTP
Server
User
Details
Database
Join the conversation â use the hashtag #risktesting on Twitter
48. Risk: Basic Acceptance Criteria is not enough
⢠A user must correctly fill in their
username and password on the website
before they login and access the system
â User Exists, Password correct
⢠user logged in
â User Exists, password wrong
⢠user not logged in
â User does not exist, password meets
valid criteria
⢠user not logged in
High Level
Acceptance
Criteria
Join the conversation â use the hashtag #risktesting on Twitter
49. Mitigate risk of missing Acceptance Criteria
⢠We would ask for additional information
about requirements and acceptance criteria.
â How often can a user try to login?
â What if user is already logged in?
â What error messages displayed?
⢠For getting password wrong
⢠When user does not exist
⢠If username blank
⢠If password blank
⢠etc.
Acceptance Criteria
Nuances & Details &
some technical
implementation
details
Join the conversation â use the hashtag #risktesting on Twitter
50. Mitigate limited coverage of business domain to
cover web page structure and platforms
⢠Non-domain input
â Username and password are text fields
⢠how much text can they handle? maxlength=â20â, JS validation
⢠Unicode chars? JS validation of valid chars
⢠Drag files in?
⢠URLs
⢠Special chars
⢠Injection payloads
⢠Etc.
⢠Platform concerns
â Browser Compatibility, JavaScript
Technical
implementation
details and
platform risks.
Join the conversation â use the hashtag #risktesting on Twitter
51. What is Technical Testing?
Testing informed by a technical
understanding of the system.
⢠Not programming. Not automating.
⢠Technical knowledge Applied toTesting
Join the conversation â use the hashtag #risktesting on Twitter
52. Letâs Build a System Technical Model
HTTP
Server
User
Details
Database
⢠failedLoginCookie & JavaScript (disable login)
⢠JS Validation of username password:
⢠Chars
⢠length
Join the conversation â use the hashtag #risktesting on Twitter
53. Technical Testing Model
⢠failedLoginCookie &
JavaScript (disable login)
⢠JS Validation of username
password:
⢠Chars
⢠length
⢠What if user disables cookies?
⢠What if user amends cookies?
⢠What if JavaScript disabled?
⢠What if JavaScript amended?
⢠What if maxlength html changed?
Join the conversation â use the hashtag #risktesting on Twitter
54. Technical Testing Skills
⢠failedLoginCookie &
JavaScript (disable
login)
⢠JS Validation of
username password:
⢠Chars
⢠length
⢠What if user disables cookies?
⢠What if user amends cookies?
⢠What if JavaScript disabled?
⢠What if JavaScript amended?
⢠What if maxlength html changed?
⢠What browser is JS compatible with?
Do we have the technical knowledge to:
⢠Spot the technical risks around reqs
⢠Identify the âwhat ifâ risks
⢠Know how to manipulate JS, HTML,
and Cookies
1. HTML
2. Cookies
3. How to disable JavaScript
4. Multiple Browsers
5. Browser Dev Tools
6. How to write JavaScript
7. Use the JavaScript Console
8. Intercept and manipulate the source
through a proxy
Join the conversation â use the hashtag #risktesting on Twitter
55. System
Model
Technical
risks.
Risk that we ignored HTTP transport layer and
server communication
What Risks are there from technical knowledge of HTTP and
Server?
⢠JavaScript and Server side validation use different rules
⢠Server side does not implement max failed logins 10 times
⢠Server side max login count is tracked separately from client
count
⢠Server side canât handle form field input values > 20
⢠âmassiveâ input values cause server to crash
⢠Invalid form details are not processed correctly
⢠Submitting form to different end point causes problem
⢠Adding basic-auth headers fools system
⢠etc.
Join the conversation â use the hashtag #risktesting on Twitter
56. Do we have the technical
knowledge to identify these risks
and build this model and explore it?
Risk that we ignored HTTP transport layer and
server communication
What Risks are there from technical knowledge of
HTTP and Server?
⢠Risk that the JavaScript and Server side
validation use different rules
⢠Risk that the server side does not implement
max failed logins 10 times
⢠Risk that the server side max login count is
tracked separately from client count
⢠Risk that server side canât handle form field
input values > 20
⢠Risk that âmassiveâ input values cause server to
crash
⢠Risk that invalid form details are not processed
correctly
⢠Risk that submitting form to different end point
causes problem
⢠Risk that adding basic-auth headers fools
system
⢠etc.
1. HTTP
2. Observe HTTP Traffic (proxies or dev tools)
3. Manipulate and send HTTP form submission
without GUI using Proxies
4. Access to server logs
5. Telnet, SSH
Join the conversation â use the hashtag #risktesting on Twitter
57. How did we get to this?
Structure
of
Technical
System
Platform
&
Input
Common
DomainReqs
Join the conversation â use the hashtag #risktesting on Twitter
58. What are the risks of doing this?
⢠we donât have the skills
⢠we donât have the inclination
⢠our staff donât want to learn
⢠we donât have the time to learn
⢠we do technical stuff and ignore the
ârequirementsâ
⢠we donât have the tools
⢠we are not allowed to use the tools
⢠we canât âsellâ this to our managers
We have to
decide if
these are
important
enough to
mitigate
Join the conversation â use the hashtag #risktesting on Twitter
59. What are the risks of not doing this?
⢠Risk that we miss entire areas of errors in our testing.
⢠Risk that no-one reviews the system at this level of technical
details.
The errors that can slip through, can be system threatening.
The easiest place to do this type of testing is through exploratory
testing.
Join the conversation â use the hashtag #risktesting on Twitter
60. How can we do this?
⢠You can use all the various mnemonics and âheuristicsâ that are
out there, to expand your analysis of the system.
â http://www.qualityperspectives.ca/resources_mnemonics.html
⢠Work from âfirst principlesâ
â Build system and technical models
â Analyse the model for gaps and risks
⢠Both require you need to increase your technical knowledge:
â To work from first principles to build a model and identify gaps in
your knowledge and identify risks
â To gain maximum value from the mnemonics because they help you
explore your model
Join the conversation â use the hashtag #risktesting on Twitter
61. Simple decisions
⢠Are you prepared to increase your technical
knowledge?
⢠Are you prepared to put in the time and effort to
learn more?
â You /Your Company /Your Manager /Your
Project
Join the conversation â use the hashtag #risktesting on Twitter
62. You donât have to know everything
If learn in small chunks, you apply what you
learn, during your testing, then you will keep
learning and keep your knowledge up to date.
Join the conversation â use the hashtag #risktesting on Twitter
63. My High Level Guide
⢠Model
â Model what you know.This will help you identify gaps.
⢠Observe
â How can you observe technical details?
⢠Reflect
â Think about gaps in the model, risks, issues, capabilities.
⢠Interrogate
â How can you drill deep into the information and system?
⢠Manipulate
â How can you interact with it at a technical level.
Join the conversation â use the hashtag #risktesting on Twitter
64. Warning: Risks
⢠You will test slower when you are learning
⢠You will be more uncertain because you are
expanding your model
⢠You might raise false flags because you
misunderstand what you are seeing
⢠You will go down rat-holes that lead nowhere
⢠You will spend time evaluating tools
Join the conversation â use the hashtag #risktesting on Twitter
65. Hints
⢠You will test slower when you are learning
â But you will speed up when you are more proficient
⢠You will be more uncertain because you are expanding your
model.You might raise false flags because you misunderstand
what you are seeing
â But you will learn to understand what you are seeing
⢠You will go down rat-holes that lead nowhere
â Time-box investigations, the same with exploratory testing
⢠You will spend time evaluating tools
â Donât evaluate them in isolation. Use them on the project.
Join the conversation â use the hashtag #risktesting on Twitter
66. Yeah, but seriously, Iâm a managerâŚ
⢠Iâm in meetings all day
⢠I nod when my staff tell me stuff
⢠If it isnât an email or a word processor or a spreadsheet, I donât
open it
Join the conversation â use the hashtag #risktesting on Twitter
67. I manage seriouslyâŚ
It is always an individualâs choice to improve their technical skills.
But a managerâs job is to manage risk.They can decide to take
action to mitigate the risk that there are gaps in testing caused
by a lack of technical focus regardless of their technical
knowledge.
Join the conversation â use the hashtag #risktesting on Twitter
68. Start to End
⢠We test systems to the level that we understand
them enough to observe their behaviour and
compare it to our model of how we think it
should behave.
⢠We test systems at the places where we can
manipulate them.
⢠We test systems to the level that we can
interrogate them to understand the data that
they process and produce.
Join the conversation â use the hashtag #risktesting on Twitter
69. End to End
Join the conversation â use the hashtag #TBD on Twitter
⢠Expanding our technical knowledge expands:
â Our models
â Our ability to observe
â Our ability to reflect on gaps and risks
â Our ability to interrogate the system
â Our ability to manipulate the system
â Our ability to test
70. End to End
⢠Expanding our technical knowledge expands:
â Our models
â Our ability to observe
â Our ability to reflect on gaps and risks
â Our ability to interrogate the system
â Our ability to manipulate the system
â Our ability to test
And the risk of not doing that, is not
one Iâm prepared to take.
Join the conversation â use the hashtag #risktesting on Twitter