Tired of doing upfront test script creation in your testing efforts? Feeling bad for demotivating your testers? Want something to replace this sickening approach to software testing? This presentation outlines why test scripts are not useful, and how test ideas are the new way forward to better testing. Coverage, traceability, reporting, automation and skills are all covered. Take a quick look and see if you can see there is another way to do software testing that is actually pure common sense.

1. Demise of Test Scripts,
Rise of Test Ideas
Richard Robinson
8 April 2013

2. Who is Richard?
● 10 year old tester
● Consultant / contractor
Blog www.hellotestworld.com
Twitter @richrichnz
Skype richrob79
● Began thinking since
Dice Game in 2009
● Co-organiser of Sydney Testers
● Background in Psych
and Computer Science
● Enjoys running, piano,
photography, travel

3. Subject Matter
"Demise of Test Scripts, Rise of Test Ideas"
What do I mean by test scripts:
● upfront analysis
● templated
● preconditions, steps, expected result,
actual result, pass/fail
● not automated

4. One of these...

5. Or these....

6. Terminology
CHARTER - Learning and execution log that captures all
related information of the feature analysis. Includes
metacognitive comments, tests, questions, bug
observations, modelling and anything else relevant to the
test session.
FEATURE - Any chunk of a system or product. Logical slice
of functionality. eg. an interface, a webpage screen, a
workflow scenario.
SILO Scratchpad for holding ideas of a common nature
such as questions, test ideas, or problems.
TEST IDEAS - Lets discuss test scripts first, then get on to
test ideas...

7. Test scripts - hands up
How do you feel about test scripts?
What do you think about when you are reading
a test script document?
frustrated!
Repetition
Repetition
Repetition
Repetitio
Repetition
Rep
OMG!

8. Cem Kaner on test scripts
● Myth that scripts teach testers the system
● In fact, they do the opposite
● Steps switch off learning, intuition,
questioning... focus on steps themselves
● Testers learn faster when they discover
concepts, rather than by being told them

9. There are books on this stuff...
Adults benefit from
activity based and
discovery-based styles of
learning, not scripted
instruction-based.
The Adult Learner
Knowles, Holton, & Swanson

10. Scripts promote confirmation bias
● Humans have selective attention
● A scripts expected result biases them to
find it, and miss everything else.
● This is known as confirmation bias.
http://en.wikipedia.org/wiki/Confirmation_bias

11. Inattentional blindness...
Test scripting focussed the mind on a strict set
of steps and expected outputs.
So what happens when something goes wrong
that is not related to the test?
Answer: blindness

12. Keith Klain on test scripts...
● Testers ignore everthing off-
script
● Testers cannot react, learn
and change their approach
● Factory model outdated
● Offshoring becoming costly
● Focus moving to skilled
testing

13. Let's do something about this...
Test scripts paint a bad picture for software
testing, quality and don't add much value to a
product or system.
Is there a
better way?

14. Introducing test ideas...
● Single words or phrases that represent a
concept to be considered for testing
● A checklist used as a guide to inspire
valuable tests relevant to the feature
● Exist in any intended granularity
● Unit of testing currency that help focus
tester on what is important

15. Here are some test ideas...
Happy path
Usability
Main flow
Alternate flow
Negative tests
Error tests
Boundary conditions
Database data
Configuration
User personas
Production records
Production systems
Privacy
Standards
Security
Reconciliation tests
Platform performance
Mobile

16. Why use test ideas?
● Act as a guide, not a control
● Antifragile and evolve with project
● Does not assume complete specifications
● Traceability and progress reporting
● Reporting on what was NOT tested
● More likely to achieve insight
● Empowers and motivates tester
● Simplified, natural way of testing

17. Compatible with how people learn
When faced with a new
feature, if a testers initial
understanding is engaged,
then the tester is more
likely to succeed in
understanding new
concepts and information.

18. But how do I use test ideas?
● Can have none, one or many tests
● Can be of any granularity that seems
relevant to the feature under test
● Can be captured in a checklist at any time
● Needs to be evaluated against the context
of the testing effort
● Should be explored from simple to complex,
alongside the learning process
● Need to be captured in the summary report
for the feature under test

19. Demo feature: login page
Your mission is to test the latest feature to be
developed, the login page.
Go test!

20. Demo of test ideas in a method...
Start here:
● Developer demo
● Repeat simple path
● Tour the feature
● Early discovery

21. Demo of test ideas in practice..
Feature introduction
Should behave like any other log in screen.
Our data is very important to us, so we should not
allow a robot to attack the log in to steal user
accounts.
We pride ourselves on appearance so usabililty
might be important.
Requires user accounts to be set up. These ones
have already been set up: - dev:password123
● Begin charter document

22. Demo of test ideas in practice..
● Begin charter document
Info sources
Myee, the developer
Compare with other log in functions such as
facebook, ebay http://ebay.com/login
Is there a standard for log in screens? Yes - found
one: http://wikipedia.com/loginstandards
Read the spec and data dictionary for detail around
workflows, found here http://corporate.wiki.
com/datadictionary

23. Demo of test ideas in practice..
● Begin charter document
Initial test ideas
Input field boundaries and
definition
Failed login attempts
Locked account
Reset account function
Request password and username
Stay signed in
Configuration
Cookies
Encryption of data
Sessions and public terminals
Multiple sessions on multiple
devices
Security
Usability - tabbing, enter to
submit?
Notifications
Error messages - different for bad
username and bad password?
Help link
Timeout
Re-captcha on reattempts

24. Demo of test ideas in practice..
● Begin charter document
Initial questions
What browsers are important?
Is mobile browsing supported?
How much time do I have to test this?
Where can a user log in from?
Images didn't display, is this a bug?
Is this functionality out of the box or custom built?
What is the name of the function?
What platform are we operating on?
What version of html and ajax are we using?
Is the page compliant with YSlow?

25. Model moving parts
Log in attempts: first time, less than max, at max
Account state: unlocked, locked
Log-in state: logged in, logged out, session expired
logged out
Time: recent session expired, first time login on this
pc, 2nd time (cookies)
Demo of test ideas in practice..
● Begin charter document

26. And the rest of the method...
● Execute some standard test ideas
● Gather more test ideas, seek more
information
● Investigate deeper test ideas
● Consider automation
● Consider performance
● Consider security
● Consider compliance
● Document tests
● Tester demo

27. Finally the coverage and report

28. Questions?

29. James Bach offers his opinion...
Play in browser

30. Method Analysis
1. Engage3. Info sources
2. Dev demo
4. Tour &
charter ideas
5. Model
6. Gather info
7. Order ideas
8. Investigate
9. Automate
10. Performance
11. Security
12. Compliance
13. Document
14. Test demo

31. Step 1: Engage
● Take early notes on the feature
● What is important?
● Why test at all?
● Start your charter

32. Step 2: Dev demo
● Best done on testers machine
● Prove that simple path works
● Prove that config ok
● Builds relations
● Early questions
● Keep it brief

33. Step 3: Identify info sources
● People
● Drivers
● Project state
● Contract?
● Vendors
● Relationship managers
● Data dictionary
● Database access
● Standards

34. Step 4: Tour and charter ideas
● System tours at your will
● Early tests
● Early learning
● Gather test ideas
● List questions
● Identify testability challenges
● Get curious!

35. Step 5: Model moving parts
● List states of things
● List transitions
● List actor types
● List groups of relevant partitions
● List key scenarios around time
● List anything that moves
● List key business rules
You may also like to jot down test cases from
simple, standard flow, negative, complex
cases

36. Step 6: Gather more info
● Read and gather more information
● Process more test ideas from it
● Refer to accronym heuristics
○ FEWHICCUPS, SFDPOT
● Ask questions
● Learn learn learn!

37. Step 7: Order your ideas
● Checkpoint
● Consider wider context
● Stop look listen
○ Time
○ Business drivers
○ Stakeholder messages
○ Contractual requirements
○ Trouble areas (developer, history, complexity)
● Order tests by complexity
○ Main flows / alternate flows
○ Error cases
○ Complex interactions

38. Investigate test ideas strength-based
● Gradually spiral outwards, from simple tests
to complex
Time
Depth
BA
Dev
Tester

39. Step 8: Begin test idea experiments!
● You can start this any time, but you should
be launching into it by now
● You are feeling confident about the feature
● Your learning is now being cashed in
● Conduct regular checkpoints
● Raise bugs
● Draft test procedures that may be reused

40. Step 9: Consider automation
Support automation
● What tests require code to execute?
● How can you improve the efficiency?
● What tools might assist the testing?
Publish to auto framework
● Propose some tests for inclusion
● Purpose is to check only, not test
● Monitoring function

41. Step 10: Consider performance
● Performance is a major concern at go live
● How does load impact your feature?
● How can you increase your volume?
● Is this consideration relevant?

42. Step 11: Consider security
● Security compliance is a major
consideration for most systems
● Sensitive data
● Encryption
● Hacking
● Basic penetration tests

43. Step 12: Consider compliance
● Non-compliance can halt a project dead
● Compliance standards
● Safety of data
● Global standards

44. Step 13: Document tests
● List out tests into specification
● Indicate which are simple, standard, and
complex tests
● Offers evidence of process for auditors
● Charters are usually messy
● Useful for identifying tests for regression

45. Step 14: Tester demo
● Tester demonstrates feature findings
● Chance to give back info discovered
● Chance to demo skills and test cases

46. Wise words...

47. Final word...
Play in browser

48. Resources
Metrics and Measurement
Metrics, Qualitative Measurement, and
Stakeholder Value
www.kaner.com/pdfs/metricsCASTkaner.pdf
Kaner, 2009
Software Engineering Metrics: What Do They
Measure and How Do We Know?
testingeducation.org/a/metrics2004.pdf
Kaner & Bond, 2004
Testing Coverage
Software Negligence and Testing Coverage
http://www.badsoftware.com/coverage.htm
Kaner, 1995
Exploratory Testing
Exploratory Testing Explained
http://www.satisfice.com/articles/et-article.pdf
Bach, 2003
Session Based Test Management
http://www.satisfice.com/articles/sbtm.pdf
Jon Bach, 2000
Test Heuristics Cheat Sheet
http://testobsessed.com/wp-
content/uploads/2011/04/testheuristicscheatsheetv1.pdf
Qualitytree.com, 2006
Mnuemonic mind map
http://www.xmind.net/embed/YvxE/
Charrett
How to Adjust to the Changing Face of Software Testing?
http://www.cio.
com/article/730131/How_to_Adjust_to_the_Changing_Face
_of_Software_Testing?page=1&taxonomyId=3040
Heusser, March, 2013
Blog: FEW HICCUPPS
http://www.developsense.com/blog/2012/07/few-hiccupps/
Bolton, July 2012

49. Appendix: Meetup responses
"What do you think of test scripts?"

50. Appendix: Charter template
CHARTER
HYP/MISSION:
TASKS:
MODEL DATA:
INITIAL QUESTIONS:
INFORMATION:
EVIDENCE LOCATION:
ACCESS:
TESTS:
SCOPE:
VERIFICATION METHODS
LEARNING TASKS
=============
DEV DEMO -
OPERATION LEVEL TESTS -
READ KEY DOCUMENTATION -
REVIEW APPROACH DESCRIPTION
MODEL MOVING PARTS
REVIEW SPECIFIC TEST IDEAS
DB Field Check -
Mandatory Fields -
Optional Fields -
Field Length Boundaries -
Field Types
Full/Partial Records
Duplicate Records
XSD Structure Check -
Validation Rules Enforced
Negative Pathway Tests
Multiple Sets of Data
Logical Flow Tests
Error Coverage -
Intent Based Tests -
Check Logging for Errors
Privacy & Encryption
Standards -
Risk Evaluation
Light Load Tests -
Race Conditions -
UNIT TESTS IN SOAPUI
SCENARIO IDENTIFICATION -
DEV QUESTION & DEMO
INTEGRATE FULLY WITH
FRAMEWORK -
DOCUMENT TEST CASES
DEMO TO TEAM
SUMMARY OF KEY FINDINGS
============
REPORT TO PM
============

51. Appendix: Skills framework
● Self-organised modules
● Written by testers for testers
● Closely related to specific test ideas
eg. DB Sql statements, bash scripts for
logging, manipulation of data in np++