Elisabeth Hendrickson’s book, Explore It!, contains this definition: “Tested = Checked + Explored”. When I read it, I was fascinated. “What does that mean?”, I asked myself, “what does it /really/ mean?”
This talk described the journey I undertook to understand it, and other definitions of testing that I found along the way, and then to come up with a new definition that filled the gaps I saw in the others, without losing the aspects of them that I felt were valid and useful.
Essentially, I formalised what testing is for me. And, now that I have my definition, I can ask myself in any given situation whether my actions are consistent with the way I believe I want to behave.
2. Testing
is the pursuit of relevant incongruity
Testing (for me)
is the pursuit of relevant incongruity
@qahiccupps
3. Tested = Checked + Explored
Neither checking nor exploring is sufficient on its own
… interact with the software or system, observe its actual
behavior, and compare that to your expectations
@qahiccupps
4. Does testing have to be on the software?
“Tested = Checked + Explored” …
… but for what values, for what units, and what does “+” mean?
Past tense suggests testing and finishing testing.
Do we always need both checking and exploring?
How does this relate to the testing/checking distinction?
@qahiccupps
6. But was I testing?
It felt like testing!
@qahiccupps
7. Some definitions of testing
“Testing is the process of executing a program with the
intent of finding errors.”
Glenford J. Meyers; The art of software testing [1979]
“The penultimate objective of testing is to gather
management information.”
Boris Beizer, Black Box Software Testing: Techniques for
Functional Testing of Software and Systems [1995]
“Software testing is the dynamic execution of software
and the comparison of the results of that execution
against a set of pre-determined criteria.”
NIST The Economic Impacts of Inadequate Infrastructure
for Software Testing [2002]
“Testing is… an infinite process of comparing the invisible
to the ambiguous in order to avoid the unthinkable
happening to the anonymous”
Jon Bach,Telling Your Testing Story, SEASPIN [2008]
“Testing is the process of evaluating a product by learning
about it through exploration and experimentation, which
includes to some degree: questioning, study, modeling,
observation, inference, etc.”
James Bach, Michael Bolton [2013]
Arborosa,
What is software testing? @qahiccupps
8. Testing is simple: you understand what is
important and then you test it
Adam Knight, Fractal Exploratory Testing
Rikard Edgren, EuroSTAR 2015
as each flaw ... is discovered ... [a] mini
exploration will result in a more targeted
testing exploration around [the] feature area
@qahiccupps
9. Testing a sub-sub-feature
Testing a sub-feature
Testing a feature
Testing the model
Testing the PO’s view
Testing the links between features
Testing another feature
Testing PO’s expression of their view
Testing the way I’m talking to the PO
Testing whether the PO is the best
person to talk to
Testing the end user need
Testing the feature testing
Testing the reason for testing
@qahiccupps
12. So what is testing about (for me)?
Often non-linear
About more than software
About potential issues
Looking and maybe finding
Takes place in a context, at a time, for a reason
Takes place to inform a decision
@qahiccupps
13. Testing
is the pursuit of relevant incongruity
Testing
is the pursuit of relevant incongruity
@qahiccupps
14. Testing is the pursuit of relevant incongruity
not in harmony or keeping
@qahiccupps
15. Testing is the pursuit of relevant incongruity
finding
exploring
investigating
@qahiccupps
16. Testing is the pursuit of relevant incongruity
to someone who matters
@qahiccupps
17. Testing
is the pursuit of relevant incongruity
Testing
is the pursuit of relevant incongruity
@qahiccupps
18. The goal of testing is identifying problems that matter
Michael Bolton, Quality Remarks podcast #10
@qahiccupps
19. Testing
is the pursuit of relevant incongruity
Testing (for me)
is the pursuit of relevant incongruity
@qahiccupps
21. Selected References
Jon Bach: https://www.seaspin.org/s/telling_your_testing_story.pdf#page=11
Rikard Edgren:http://qahiccupps.blogspot.co.uk/2015/11/testing-is-simple-and-complicated.html
Adam Knight: http://www.a-sisyphean-task.com/2013/01/fractal-exploratory-testing.html
Arborosa: https://arborosa.org/2015/06/14/what-is-testing/
Quality Remarks: http://qualityremarks.com/qr-podcast/
Explore It!: https://pragprog.com/book/ehxta/explore-it
22. 22
Some Comments
Helicopter view
• it is sufficiently open that people could buy into it, and read into it, particularly non-testers.
• it's accurate and to the point.
• it has the feel of Weinberg's definition of a problem.
• it sounds profound but I'm not sure whether there is any depth.
• it seems very close to the regular notion of targeting information/unknowns.
Coverage
• can not testing be part of this idea of testing?
• Cem Kaner talks about balancing freedom and responsibility in testing. Is that covered here?
• the definition doesn't talk about risk.
Practical utility
• it couldn't be used to help someone new to testing decide what to do when testing.
• I could imagine putting this onto a sticky and trying to align my actions with it.
Definitional
• incongruity is too complex a word.
• what other words could replace testing in the definition and it still hold?