EuroSTAR Software Testing Conference 2013 presentation on Software Testing Secrets We Dare Not Tell by Graham Thomas. See more at: http://conference.eurostarsoftwaretesting.com/past-presentations/

1. Software Testing Secrets
That We Dare Not Tell !
Graham Thomas
Independent Software Testing Consultant
www.eurostarconferences.com
@esconfs
#esconfs

2. Abstract
I have worked in IT for over 30 years, in software testing for
over 20 years, so you might think that I know a lot about IT
and in particular software testing. Well, I am going to share
some dark secrets about software testing with you, which we
dare not tell, and do that in the form of questions.
These questions are important because they drive at the
very heart of what we are doing in the software testing
industry today.
2
This session has been designed to be a highly interactive
discussion which many people might find challenges their basic
understandings. I will act as facilitator, give an introduction to
each question, then actively moderate the debate and if needed
act as arbiter. Come along, expect to be involved, and if you have
a view then please share it. Help to drive forward the discussion –
and the software testing industry.

3. A bit about you
A feel for the audience today
 Current Role
 Above Program Test Manager
 Program Test Manager or Test
Manager
 Test Leader
 Test Analyst or Test Execution
 New to Testing
 Industry Experience
 15 years +
 Over 10
 Over 5
 Done a year or more
 Still got the L plates on
 Industry exposure
 Industry body
 Workgroup
 Contributor
 Blog
 Speaker
 Author
 Conferences
 First time at EuroSTAR
 Other conferences and seminars
 Workshops or meetings
3

4. Secretive Air Force Space Plane X37B
Lands
4

5. 1. What is the purpose of
Software Testing?
5

6. What non-testers think
I was explaining this presentation to my doctor
the other day.
I said “It is questioning the purpose of Software
Testing.”
He said “ That is easy, it is to find all the bugs.”
I said “Not necessarily.”
He gave me a funny look, and I realised that this
was a longer conversation that we had time for!
6

7. 7

8. 8
2. Just how effective is the way we test –
and how do we know? (Trad, V-Model,
Structured Testing, agile or any other
form of testing for that matter?)

9. Effectiveness
 GBP / USD per bug
 Overall cost by requirement / use case /story
 Organisational cost – If I do this I can’t then do that
 Can I afford to test?
 How long will it take
 Why is it now taking longer
 How do I measure it?
 Can’t I have it now?
 And what will I get for all that
 The Time / Cost / Quality Dice
 How will I know I have got all of that?
 Do I need all that?
 What happens if I don’t get all of that?
9

10. Discuss
10

11. 3. If checking isn't software testing, then
why is it that ‘checking’ is what our
stakeholders are paying us to do?
11

12. Drivers for Testing
Checking
 Prove it works
 Show me it working
 Coverage
 Understand what I will get
 Quantify the risk of delivery / non-delivery
Testing
 Michael Bolton quoting Laurent
Bossavit
“A Testers Role is to see through the veil of
conflicting illusions!”
 Agile Testing is ‘Exploratory Learning’
 Discovery
12

13. Discuss
13

14. 4. If software testing is so
difficult, demanding and
challenging, then why is it
that we keep on assigning
the least skilled or
experienced to perform it?
14

15. Conundrum
And even if we don’t know;
 what the purpose of software
testing is,
 how effective our software
testing is,
 or agree on how we go about
software testing,
surely we should at least be able to
draw together some attributes for
the kind of people we need to
perform software testing?
15

16. Discuss
16

17. 5. Why do software testers
spend so much of their time
running tests that do not find
bugs?
17
Tools Scripting Execution Analysis Resources

18. Inefficiencies
 Automation Paradox
 To successfully automate tests they all have to pass. However what is
the point of running tests that don’t find any bugs?
 Tools to automate testing
 If we had tools that could test software we wouldn’t need software
testers!
 Though, how would we know whether the tools were working or not?
 Automating testing does not save any time, and in most cases
takes longer than manual testing!
 The latest trends in test optimisation are to stop running
regression suites that take time effort and resource but no longer
find bugs!
18

19. Discuss
19

20. To Conclude
As an industry, collectively:
“We do not know, or agree on the purpose of software testing”
“We do not know how efficient or effective our test models are, regardless of type”
“We don’t even agree on what it is we should do when we are testing software”
“We don’t understand what skills and experience we need to perform software testing”
“And finally when we do testing, an awful lot of it doesn’t actually make any difference”
I think we should either:
“Keep these secrets very, very quiet”
“Or tell everyone . . . . . .
and set about finding the answers!”
20

21. Let me think about that for a
minute
21

22. Contact Details
Graham Thomas
Independent Software
Testing Consultant
 graham@badgerscroft.com
 +44 7973 387 853
 www.badgerscroft.com
22