A brief introduction to test for the non-tester. Can be used for both business and development, although it is primarily focused on developers and persons interested in becoming testers.

1. Introduction to test:
What is test, and who is the tester?
•Mattias Lönnqvist
•Kuoni Nordic IT
•STOCKHOLM, 2013-11-27

2. Agenda
• Presentation
• Introduction to testing
•
•
•
•
•
What is testing?
Why do we test?
Test Phases
Requirements
When do we stop?
• Who is the tester?
•
•
•
•
•
Beizer
Cognitive psychology
Biological differences
Assumptions
Cognitive dissonance
•
•
•
•
• Is all lost?
• <- Briefly!
Why does it matter?
Test tools
Summary
Q&A
|
1

3. Presentation
• Mattias Lönnqvist
• QA Test Manager at Kuoni Nordic
IT (Previously: OE IT Online)
• Been at Kuoni since June 1st 2012
• Been working in IT since 1997 and
with test since 2004.
• ISTQB certified at foundation level
and advanced level (test manager)
• Worked as a consultant in test for 5
years before going to Kuoni
• Will give you a quick background
on testing
• Please do interrupt if you have
questions
• A nice guy 
| 2

4. What is testing? Introduction
• What testing is NOT
• What testing IS
• It’s not the quality process
• ”Testing is measuring quality”
• Only quality measurement, (Ariane 5 Flight 501) Hetzel 1988
• William
Ariane 5's first test flight
i.e. not theJune 1996 failed, with the rocket selfon 4 Quality process
destructing 37 seconds after launch because
of a malfunction in the control software.
• It’s not a guarantee
• ”Testing is the execution of
• Products will never be
software in order to show that it
bugfre A data conversion from 64-bit floating
doesn’t work”
point value to 16-bit signed integer value to
• Glenford Myers 1979
be stored in a variable representing horizontal
bias caused a processor trap (operand error)
• It’s not a cost
because the floating point value • Example: The Ariane 5 rocket
was too large
• We save more on testing a 16-bit signed •integer. $800 million
Cost:
to be represented by
than the cost
• It’s not a waste of time
• It helps improving quality
| 3

5. What is testing? From a business perspective
Cos
t
Reduced Cost = reduced
Quality
Reduced Time = reduced
Busines
Quality (or increased cost)
s
Increased Quality = increased
Oriente
cost/time d
Quality
Tim
e
Testing
| 4

6. What is testing: The V-model / the W-model
| 5

7. Why do we test? Introduction
• We are measuring quality
• Testing is gaining knowledge
• To keep our deadlines
• We cant keep our deadline if we dont know if it works
• Not loosing money
• Ariane 5 rocket: $ 800.000.000
• Heathrow terminal 5: £25.000.000 (failed luggage handling)
• Forsmark 2006: 2 (of 4) backup generators didn’t start on power
shortage. Cost unknown.
• etc
• To make our customers happy
• We need them to have confidence in us
• Errare humanum est
• Code is written by humans, and humans make mistakes
• Deadlines causes stress and stress induces errors
• Changing development and maintenance environments
• Lacking or incorrect requirements
| 6

8. Why do we test? The numbers
• Standish group 1995 – The Chaos Report
• 31,1% of all (software) projects are never completed
• 52,7% of all projects cost 189% of estimated cost
• In 1995, in the US, 81 billions were spent on projects that were cancelled
or never finished
• 59 billions were spent on projects that were finished that year
• Completed projects had in average 42% of initial requirements met
• 9% of all projects in larger companies were considered succesfull
• The biggest reason for this was incomplete requirements
• The numbers are better now, but not good
• Standish report costs money, which is why I use the 95 report
• Info-tech research group 2006
• 50% of additional work in software projects is caused by problems with
requirements
• 70% of failed projects fails because of problems with requirements
• Testing is connected to requirements!
| 7

9. Why do we test – Requirements
•
•
•
•
•
•
•
Verification of requirements
Business specifies requirements
Development transforms requirement into code
Testing verifies code against requirements
Requirements can never cover 100%
Requirements can never be unambiguous
Requirements can be misunderstood
•
•
•
•
•
•
•
•
•
What is this? [Example on requirements]
1. Carries objects or people
2. Low maintenance
3. Low cost
4. Versatile
The developers interpret as:
System testers interpret as:
Acceptance testers interpret as:
No! This is what business meant:
Myth: I
can’t
determine
if the code
is correct
Truth: Yes,
you can!
| 8

10. Why do we test: the costs
• What are the costs?
• Test environment
• Test resources
• Time spent on testing
• Etc
• What are the benefits?
• Compare to cost of cancelling a release
• Compare to cost of fixing a bug in live environment
• Compare to cost of decreased sales
• Compare to cost of reduction in Brand value
Source: Barry Boehm 2007
| 9

11. Why do we test: The importance
• How bad is a bug?
• Disaster?
• Titanic, Ariane, Patriot Missile, Landing system, etc
• Scale of disaster is proportional to the belief it cannot fail
• Sometimes called ”Famous last words”
• Bad?
• Word 97 crashed all the time in first release
• Software wipes database
• Trivial?
• Something is ugly, spelling error, etc.
• Catch 22
• If you look for defects and find none you are pleased
• Finding errors destroys confidence
• Testing is looking for errors you don’t wish to find…
| 10

12. Why do we test – Summary
Statement
•
•
•
•
•
•
•
•
•
•
•
True or False?
Because Software contains bugs/errors?
To estimate / measure quality in Software?
To spend time between code complete and go live?
To prove that there are no bugs?
Because testing is a part of software development?
Because Errors can cost a lot?
To reduce risk of upset customers?
To reduce risk of loosing business?
To stay the best in the travel business?
Because we like to point out others mistakes?
To be able to determine if the software can go live?
| 11

13. The test phases – simplified description
• Developers:
• Develop code
• Compile
• Check in
• Unit test
• (sometimes) integration test
• Own testers
• Smoke test
• System test
• Regression test
• Re-test
• Customer
• Acceptence test
• This is the customer sign-off, stating if they are satisfied
• No acceptance test = we accept the quality, whatever it is…
| 12

14. The test phases – One (of many) way to split up the work
• Unit tests
• Developers test their new code
• Does it compile? Does it run?
• System test
• Developers make sure system works after new code is added
• Smoke test
• Developers make sure that testers can do their work before handing over
• System Integration Test (SIT)
• Testers make sure the system is working
• Testers make sure the system is working with all other systems
• User Acceptance Test (UAT)
• Customer (might be internal) makes sure the delivered software matches
their requirements
• This phase should only find bugs relating to requirements (mismatch)
• All other bugs should be found in the previous phases!
| 13

15. Test-phases in a waterfall world (but can be described Agile)
Start
We have a project that needs testing
Requirements
Requirements
Insufficient
resources,
scope, etc
Plannin
g
Create test plan, allocate resources
Requirements
Execute test cases
Specificati
Requirements
Non-functional requirements
Create test cases from requirements
Unclear
requirement
Retest
bugs
Unclear or
missing
requirements
on
Functional requirements
A good test case is:
Execute test cases
Testin
1. Effective (finds bugs)
2. g Examplatory (covers several areas)
3. Maintainable (easy to change / update / keep current)
4. Low cost (easy to use, doesnt take a lot of time)
Requirements
Report bugs
Result
5. Possible to automate (sometimes)
s
Repor
ts
End
Create test report Requirements
Requirements
Project delivered
| 14

16. Requirements – why are they so hard?
• We are dealing
with humans
• Humans
interpret
• Humans are
different
• We have different
• Roles
• Interpretations
| 15

17. When do we stop testing?
Statement about end of testing
True or False?
• We can never test enough
• We stop testing when the customer is satisfied
• We stop testing when we have proved that the
system has no flaws / errors / bugs
• We stop testing when we ”feel” quality is good
enough
• Whether we are done depends on …
•
•
•
•
Who decides if exit-criteria is fullfilled
What are our risks
What is our test case coverage
What is our S-curve
| 16

18. When do we stop: The S-curve
This example is for a typical project
Applies to all projects
Applies to all models
Answers:
• Can we
release now?
• How close to
”ready for
delivery” are
we?
• How are we
progressing?
Risk Decision Point
deploy now?
Law of
Diminshing
Returns
Low Detection Rate
Cumulative Defects
•
•
•
•
Test on or
implement now
Undetermined
time saving
High Detection Rate
Time
Requirements
Analysis
Design
Code
Test
Implementation date
| 17

19. Who is the tester?
• We are dealing with humans
• We are controlled by our brains
• Cognitive psychology
• How / what we percieve
• How / what we see
• How / what we hear
• Test Psychology
• Understanding how we, as
humans affect testing
• We are not machines
| 18

20. What is a tester?
• Beizers phases of testing (Boris Beizer)
• Phase 0 : No difference between testing and debugging
• i.e. Testing is something done by developers only
• Phase 1: The purpose of testing is to show that the Software works
• i.e. We can prove that the software is flawless
• Phase 2: The purpose of testing is to show that the Software doesn’t work
• i.e. Our purpose is to find mistakes
• Phase 3: Purpose of testing is to reduce risks in software
• i.e. We reduce risk of errors in delivered software
• Phase 4: Testing is not an act, it is a mental discipline that results in lowrisk software without much testing effort
• i.e. testing is a part of everything
• This is what exploratory testing focuses on
| 19

21. Cognitive Psychology
• My focus is on human memory functions
• How we learn
• How we remember
• How we recall information
• Human memory consists of:
• Short term memory – here and now
• That is why you should take notes when testing
• Or you will forget what the bug was
• (But you may still remember that you found a bug)
• Medium term memory – last month
• Repeating copies from short term memory
• Long term memory – childhood (etc)
| 20

22. Human memory – Short term memory
•
•
•
•
You can only remember 7 +/- 2 items (5 to 9)
Needs to be actively copied to long term memory
Trivial errors often caused by short term memory overload
Major problem with short term memory
• I forgot what the problem is
• 
• If your brilliant idea isnt copied to long term memory you
loose it (but may come up with it again)
• That is why you should take notes when testing
• Or you will forget what the bug was
• (But you may still remember that you found a bug)
| 21

23. Human memory – Medium term memory (working memory)
• Working memory
• is more a process than a part of the brain
• Analysis causes tension
• The mind must solve the problem to relieve the tension
• Result is ”feel good factor”
• Major problem with working memory
• ”Feel good factor” undermines correct understanding
• Desire to solve leads humans to jump to conclusions
• Solution may be wrong
• E.g. When finding a work-around to do something
that isnt working the way it should, we may be too
pleased with our solution to remember the bug
| 22

24. Human memory – Long term memory
• Affects
Your world
• The way you view things
Your childhood
• Your personality
Right and Wrong
• Your temperament
Your parents and family
• The way the ”penny
life
drops”
• Education
• Familiar concepts
• Formative years
• Easily accepted
• You never forget anything
in the long term memory • New ideas & changes
• Resisted
• But you might not be
able to recall it
•
•
•
•
| 23

25. Biological differences
• Biological differences in GENERAL
• Not all men, and not all women
• Women are better at multitasking
• Women have a smaller brain, but a more developed
brain stem (the brains information highway)
• Less women has color blindness
• Important if you use color to carry information
• Men are better AND worse at math
• Women more in the middle on Bell curve
• Some biological differences enhanced by society
• E.g. high pitched voices in Japanese women
| 24

26. Assumptions
• The brain makes
assumptions from stored
knowledge
• The brain will find a
solution
• But it doesnt have to be
correct
• The penny drops …
• Do you know what you
see?
• Or do you see what
you know?
| 25

27. Assumptions – an example
• Read the following colors and say them loud
• Dont think; time is money
• We have a deadline
• Ready?
| 26

28. | 27

29. Assumptions – Contradicting associations
• Color and word doesn’t match
• Asking your brain to associate the wrong
color with the name of the color causes a
conflict
• The conflict creates overload that increases
due to quick slide change (0,5 seconds)
• Cognitive dissonance (more later)
• Now take a look at the next slide…
| 28

30. Raise your hand
if you spotted the
error…
| 29

31. • Let us look again …
• See it?
| 30

32. Assumptions - Blinded by the grammar rules
• English grammar is a set of rules
• You know the rules and apply them by
”motor skill”
• Your brain is conditioned to apply the rules,
even if what is present breaks the rules
• As a result:
• You may not ”see” what is actually
present
| 31

33. Cognitive dissonance
• Do you believe what you see?
• Or
• Do you see what you believe?
| 32

34. Cognitive dissonance - example 1(color blindness)
| 33

35. Cognitive dissonance - example 2 (it shouldn’t move)
| 34

36. Cognitive dissonance - example 3 (might not work on projector)
| 35

37. Cognitive dissonance - summary
• About 15% of all males are color blind
• Around 1% of all women
• Females can interpret patterns more
consistently then men
• If the information is in a color or if it can be
interpreted in more than one way …
• How do you know that you will interpret it
correctly
| 36

38. Interpretations
• I am a tester
• In what field?
• What do you do?
• Functional or nonfunctional?
• Manual or automated?
• Etc
• What is obvious to you might not be
obvious to other
• The better you know an area, the more
informed questions can you ask
| 37

39. Who is the tester? Summary
• To understand the tester we need to understand
•
•
•
•
•
Cognitive Psychology (memory)
Biological differences
Assumptions
Cognitive dissonance
Interpretations
• Why does it all matter?
| 38

40. Why does it all matter - Overload
• Maximum efficiency at 7 +/- 2 tasks
• Overload affects recall (memory)
• You have not only forgotten
• But you cant remember that you forgot
• Test items are overlooked
• Risk is introduced
• Impairs ability to interpret (correctly)
• You may not be self aware of overload
| 39

41. Why does it all matter – Cognitive dissonance
• You’ll remove 30-40% of your own errors
• Which is why coders shouldn’t test their
own code
• Don’t proof read your own work
• Dont design tests for software you have
written yourself
• Let others challenge your mind set, your
work and your documents
• Welcome this!
| 40

42. Is all lost ?
• What can I do about this?
• Be aware of your limitations
• Be aware of overload
• Create your own fortress of solitude:
• Set the alarm for 60 minutes and:
• Turn off the phone
• Put on head phones
• Put up a ”Do not disturb” sign
• Dont run the same test twice
• Switch with someone else
| 41

43. Test tools (in brief)
• Screen dumps
• An image says more than 1000 words
• Test case tools
• To store test cases for easy access
• TFS / Spira / Excel / etc
• With or without Gherkin / Specflow
• Bug reporting
• TFS / Spira / Google docs / etc
• Methods
• Exploratory
• Sessions
| 42

44. Summary
• Remember that testing is PART of the quality process
• Be aware of your limitations
• Both your own and those of others
• Dont expect testers to find bugs if they cant focus
• Do you know what you see or do you see what you know?
• Utilize domain knowledge (business, end users, etc)
• No one knows better than them what the software is supposed
to do!
• Not me, not the project manager, not the developers
• This is the strength and purpose of UAT!
• Avoid overload!
• Find a way to create undisturbed working space
• Humans interpret
• If you are uncertain about a requirement, ask …
| 43

45. Q&A
• Questions?
• Please do ask…
• There are no stupid questions
• Only stupid lecturers … 
| 44