Talk delivered by Craig Smith at Agile 2013 in Nashville, USA on 8 August 2013. Automated software testing is a key enabler for teams wanting to build high quality software that can be progressively enhanced and continuously released. To ensure development practices are sustainable, automated testing must be treated as a first-class citizen and not all approaches are created equal. Some approaches can accumulate technical debt, cause duplication of effort and even team dysfunctions. The seven deadly sins of automated software testing are a set of common anti-patterns that have been found to erode the value of automated testing resulting in long term maintenance issues and ultimately affecting the ability of development teams to respond to change and continuously deliver. Taking the classic seven sins (Gluttony, Sloth, Lust, Envy, Rage, Pride, Greed) as they might be applied to test automation we will discuss how to identify each automated sin and more importantly provide guidance on recommended solutions and how to avoid them in the first place.

1. 7 Deadly Sins
of Agile
Software Test
Automation
Craig Smith

2. Welcome…
Image: © Gracie Films / 20th Century Fox Television http://www.paleodietrecipes.org.uk/paleo-images/content/homer-confession.jpg

3. Adrian Smith
@adrianlsmith

4. Geeks Hate Repetition

5. It Started
With…
Image: © New Line Cinema http://www.imfdb.org/images/5/5e/SevenCover.jpg
© United Plankton Pictures https://fbcdn-sphotos-h-a.akamaihd.net/hphotos-ak-ash4/p480x480/429897_565831523456105_395040135_n.jpg

6. Discussion:
What is
stopping you
from being
great at
automation?
Image: http://toworkandback.com/wp-content/uploads/2012/08/stop-sign.jpg

7. Image: http://scitascienda.files.wordpress.com/2013/04/barney-stinson-quote-when-i-get-sad-i-stop-being-sad-and-be-awesome-instead-true-story.jpg?w=500&h=353

8. Image: http://www.cupojoy.com/images/image/Quality%20Inn%20Exterior.jpg
Building Quality In

9. AUTOMATED
TESTING
Automated
Testing
Image: http://www.lastwordonnothing.com/wp-content/uploads/2011/12/iStock_000007001634Small.jpg

10. Envy
Flawed comparison of manual testing & automation
Image: http://workingtropes.lmc.gatech.edu/wiki/images/Man-vs-machine.jpg

11. How Management
See Testing
Image: http://www.craigsfire.com/wp-content/uploads/2010/02/pool.jpg

12. How Management
Would Like To See
Testing
Image: http://us.123rf.com/400wm/400/400/dotshock/dotshock1012/dotshock101200006/8437158-internet-network-server-room-with-computers-racks-and-digital-receiver-for-digital-tv.jpg

13. Repeat After Me…
Agile
≠
Automated Testing

14. Software is a Series of Loops
Image: http://2.bp.blogspot.com/_4G2ptSlZZn0/R9nRjimkLII/AAAAAAAAAqA/tzYJjRyhLkk/s400/brekky05011049.jpg

15.  End to end automated tests and cover 10 conditions per
test, manual covers 1 condition per test
 Automated tests cover all high priority business flows,
manual tests mostly cover lower priority flows
 Metrics need context!
% Regression test cases automated
Automated Manual
Discussion:
How Much
Automated
Testing Is
Enough?

16. Manual vs Automated…
A Flawed Comparison
Image: http://fvhs.com/wp-content/uploads/2012/11/Screen-shot-2012-11-15-at-7.49.37-AM.png

17. Testing is more than a series
of merely repeatable actions
Image: http://cdn.themis-media.com/media/global/images/library/deriv/71/71335.jpg

18. Look
Left…
Look
Right…

19. Image: http://3.bp.blogspot.com/_RdF7S0TNAHY/TOqn8MLYajI/AAAAAAAAAJI/QDBFdixiq0E/s1600/2006-06-17%252520Falling-piano.gif
Look
Up!

20. 21
Image: © Jerry Bruckheimer Television http://images2.fanpop.com/images/photos/4700000/CSI-Las-Vegas-9x18-Mascara-csi-4772098-2000-1333.jpg
CSI:
Software

21.  Regression testing –
assessing current state
 Automation of test support
activities
Ideal Automation Targets
 Data generation / sub-setting
 Load generation
 Non-functional testing -
performance, security, ...)
 Deterministic problems
 Big data problems
Image: http://media.merchantcircle.com/30134145/mobile%20target_full.png

22. Common “Envy” Symptoms
 Relying on automation as
the basis for all testing
activities
 All tests are built by
developers
 Absence of code reviews
 Absence of exploratory
testing
 Absence of user testing
Image: http://snappedshot.com/turbo/sites/default/files/styles/large/public/robot-doctor.JPG

23. Approach: Desired Role
Automation:
I work with developers to
automate tests that
provide business value
and identify system risks
Collaboration:
I work with analysts and SMEs to
ensure that testable acceptance
criteria are created for all stories
Strategy & Planning:
I am involved in the project at all
stages to ensure that testing
provides the greatest value and
quality objectives are achieved
Tools & Techniques:
I use both manual and
automated techniques using
the preferred testing tools
based on the situation
Architecture:
I have an understanding of the
system architecture and can create
tests that verify individual
components and the system as a
whole
Development:
I take an interest in
development practices
and monitor code quality
metrics
Estimating:
I am involved in developing
estimates for projects at a story
level, for tools and
infrastructure and also for
deployment/release activities
Reporting:
I provide metrics that give
insight into project health
and system quality
Agile:
I understand Agile project
delivery and the differences
between testing in the different
phases of an phase
Qualifications & Training:
I have recognised qualifications
in testing and continually
update and maintain my skills
Recruitment & Development:
I am attracted to the organisation
because testing is a cool career path
that offers heaps of opportunities
and a way of continually developing
my skills
Community & Teams:
I am part of a community
of testers that are
embedded within teams
but share common values

24. Approach: Report
on Confidence

25. “Envy” Lessons Learned
 Avoid comparison between
manual and automated
testing - both are needed
 Distinguish between the
automation and the process
that is being automated
 Use automation to provide a
baseline
 Use automation in
conjunction with manual
techniques
Image: http://idahoptv.org/dialogue4kids/images/season10/robots/bookbot.gif

26. •Over indulging on commercial test tools
Gluttony
Over indulging on commercial test tools…
Image: © Universal Pictures http://static.ifood.tv/files/external-images/photo-26589.jpg

27. Discussion: What Automated
Tools Do You Use & Why?
Image: http://www.newnorfolkmitre10.com.au/pics/66/66718b.jpg

28. Some vendors
believe they
have the
“secret
sauce”
for
automation
Image: http://topnews.ae/images/HP-Sauce.jpg

29. Promise of Automation
Image: http://thebsreport.files.wordpress.com/2009/10/traveling-salesmen-medicine-show.jpg?w=500&h=364

30. Image: http://www.melhotornot.com/wp-content/uploads/Rowboat.jpg
Commercial Restricts Usage

31. Justifying The Expense
I have spent
a lot of
money on
this tool, we
better be
using it for
everything!
Image: © Warner Bros Television Distribution / Screen Gems http://3.bp.blogspot.com/-1L3LVOu0cZg/UPs4gju7DJI/AAAAAAAANx0/mrdx5RYSSW0/s1600/jetsons_L74.jpg

32.  Underlying commercial tools
technology often not compatible
with development tool chain
Incompatible Technology
 Special file formats or
databases
 Lack of version control for tests
and/or cannot be versioned
within the software
 Not easily combined with
Continuous Integration
Image: http://virtualmarketingofficer.com/wp-content/uploads/2009/06/square-peg.jpg
 Not easily adapted or extended
by the developers

33. Common “Gluttony” Symptoms
 A commercial tool forms
the basis of a testing
strategy
 Only certain teams or
individuals can
access a tool or run tests
 Developers have not been
consulted in the selection
of a testing tools
 “We always use <insert
tool-name> for testing!”
Image: http://resources2.news.com.au/images/2008/09/02/va1237328073953/burger-6230645.gif

34. Approach: Tools Guide

35. Approach: Craftsmanship
Image: © Wild Dancer Productions / Touchstone Television http://www.tvacres.com/images/props_tool_time_binford.jpg

36. “Gluttony” Lessons Learned
 Favour open software tools
where ever possible
 Use tools that can easily be
supported by the
development team and play
nicely with existing
development tool chain
 Ensure any commercial
tools can be executed in a
command-line mode so it
can be automated
Image: https://fbcdn-sphotos-b-a.akamaihd.net/hphotos-ak-prn1/p480x480/9346_506859889386718_1005343687_n.jpg
 Educate!

37. •User interface forms the basis
for all testing
Lust
User interface forms the basis for all testing…
Image: http://stuficionado.net/wp-content/uploads/2012/11/computer_hug.jpeg

38. Traditionally Test via the UI
Image: http://stovallorganizing.com/wp-content/uploads/2013/04/Turtleing.jpg

39. Manual
Exploratory
Collaboratively
built around
system behaviour
Developer built
optimised for fast
feedback
Confidence
Speed/Feedback
Exercises
components and
systems
Investment Profile

40. Understand The Architecture
Image: http://harrison.kyschools.us/hcms/TechProjects2011.htm

41. Test Design

42. F Fast
I Isolated
R Repeatable
S Self Verifying
T Timely
FIRST Test Properties
Image: http://www.moto-choice.com/PRImages/EditorImages/148330-bathurst-finish-2.jpg

43. Common “Lust” Symptoms
 Testers cannot draw the
application or system
architecture
 Large proportion of tests
are run through the UI
 Testers have limited
technical skills
 No collaboration with
developers
 Intent of tests is unclear
Image: http://snappedshot.com/turbo/sites/default/files/styles/large/public/robot-doctor.JPG

44. Discussion: What
technologies do you need to
understand to automate?
Image: © Paramount Pictures http://americanobjective.com/wp-content/uploads/2012/04/original.jpg

45. Agile Testing
Command Line
Interface
Continuous
Integration
Version Control
Build Tools
Capture Replay
(Selenium)
HTML
Test
Maintenance &
Data
Specification By
Example / ATDD
/ BDD
Concordion /
Cucumber
SQL Web Services
Approach: Test
Automation Course

46. Approach: Use
UI Tools for
Right Purpose
Use tools like Selenium
sparingly, good for
exploratory testing
Only use tools like
HTMLUnit or scrape
screens when no other
options available

47. “Lust” Lessons Learned
 Limit the investment in
automated tests that are
executed through the user
interface
 Collaborate with developers
 Focus investment in
automation at lowest
possible level with clear test
intent
 Ensure automation gives
fast feedback
Image: http://foodcandy.com/wp-content/uploads/2013/03/flowers-and-chocolate1.jpg

48. Pride
Too proud to
collaborate
when creating
tests
Pride
Too proud to collaborate when creating tests…
Image: http://latimesblogs.latimes.com/.a/6a00d8341c630a53ef0120a955baf0970b-600wi

49. Poor
Collaboration
= Poor Tests
Image: http://2.bp.blogspot.com/-RYkFdryP-Co/Tj7-0q3DyPI/AAAAAAAABJU/6tETrv8zyb4/s1600/OpsSilo.jpg

50. Poor
Collaboration
or
Dedicated
Roles
=
Duplication
Image: © Universal Pictures http://cdn3.whatculture.com/wp-content/uploads/2013/03/twins-poster.jpg

51. No
Definition
of
Quality
Image: https://lh3.googleusercontent.com/-46OEX5ffUX4/TtVNoEnFPmI/AAAAAAAADBo/g55cQRy5im0/s800/zimogs_kvalitates_zime.jpg

52. Discussion:
What is
something
you regard
as quality?
Image: http://farm7.staticflickr.com/6115/6287632596_5cc5a681c8_z.jpg

53. Analyst /
Customer
Developer Tester
Automation
Elaboration /
Specification
Acceptance
Criteria
High
Performing
AGILE
Project
Manager
Good
Collaboration

54. Developer
Tester
Analyst
More
Technical
Less
Technical
Code
Design
UnitTest
Automated
Functional/
Specialist
Tests
Manual
Test
Exploratory
Test
Requirements
Customer
Collab.
Developers need
more testing involvement
Testers need more
technical involvement
Analysts need more
testing involvement
Skills

55. Red
GreenRefactor
Red
GreenRefactor
ATDD
ATDD
TDD

56. Image: http://concordion.org/image/example/online-shop/AcceptanceTest.png
Specification
by Example

57. Common “Pride” Symptoms
 Automated tests are being
built in isolation from team
 Intent of tests is unclear
or not matched to quality
 Poor automation design
(abstraction,
encapsulation, ...)
 Maintainability or
compatibility issues
Image: http://farm8.staticflickr.com/7120/7683912508_b23c4c9ff0_b.jpg

58. Approach:
Defining Quality
Quality
Advocates
What does quality mean
to the different roles in
the team?
Quality
Taxonomy
What are quality
attributes?
Quality
Prioritisation
How do we know what
quality attributes to
include?
Quality
Tradeoff Risks
What are risks with the
quality attributes we
are trading off?
Quality
Measurement
How do we test and
measure quality?
Success
Sliders
How does quality relate
to the sliders?
Quality
Definition
What does quality
mean?
Next Steps How do we apply
quality to our work
1
2
3
4
5
6
7
8

59. Approach:
Disband Testing
Teams
Image: http://prince2pm.files.wordpress.com/2012/01/industry-groups-product-groups.jpg

60. Approach:
Simple
Strategy

61. “Pride” Lessons Learned
 Collaborate to create good
tests and avoid duplication
 Limit the investment in UI
based automated tests
 Collaborate with developers
to ensure good technical
practices (encapsulation,
abstraction, reuse, ... )
 Test code = Production
code
Image: https://c479107.ssl.cf2.rackcdn.com/files/7178/article/width668/rfwyhksh-1327469166-1327469251.jpg

62. Sloth
Too lazy to properly maintain automated tests…
Image: http://photos.pcpro.co.uk/blogs/wp-content/uploads/2011/06/Asleep-at-computer.jpg

63. New
Feature
System
Interface
Change
OS
Patch
Reference
Data
Changes
Time
Many Causes of
Automated Test Failure

64. Time
Cost/Effort
Potential
Value of
Maintained
Automated Test
Suite
Value of
Unmaintained
Automated Test
Suite
Manual test execution
Maintained automation
Unmaintained automation
Importance of
Maintenance

65. Discussion:
How often
are you
red?
How do you
know?
Who cares?
Image: http://www.eriding.net/media/photos/environment/traffic/070328_jbean_mp_environment_traffic_0135.jpg

66. Continuous Integration
Watch
Code
Build /
Compile
Run
Tests /
Analysis
Publish
Results
Stop
The
Line!
Failure?

67. Continuous Delivery
Image: http://www.thoughtworks.com/imgs/drawing-cd-header.png

68. #notesting
Image: http://3.bp.blogspot.com/-jy4Sc0m-oTA/UU9qrGJw56I/AAAAAAAAAC8/zSlPJS9Ed7c/s200/No+testing.JPG

69. Common “Sloth” Symptoms
 Test suite has not been recently
run - state is unknown
 Continuous Integration history
shows consistent failures
following
development changes / release
 Test suite requires manual
intervention
 Duplication within automation
code
Image: http://commons.wikimedia.org/wiki/File:Cairns_Australia_Beer_Run.jpg
 Small changes trigger a
cascade of failures

70. Approach:
Visible
Status /
Stop The
Line

71. “Sloth” Lessons Learned
 Ensure automated tests are
executed using a Continuous
Integration environment
 Ensure tests are always
runnable - even if the system in
not being actively developed
 Make test results visible -
create transparency of system
health
 Ensure collaboration between
developers and testers
Image: http://1.bp.blogspot.com/_yEAvQXpDW6I/S-8HgogJWpI/AAAAAAAAAY0/phpZuCfP610/s400/Linea%2520Berocca.jpg

72. Rage
Frustration with slow, brittle or unreliable tests…
Image: © 20th Century Fox http://kei.dizkartes.nl/static/img/_common/header_404.jpg

73. Fast
Feedback Image: http://www.digitallanding.com/wp-content/uploads/2012/07/Faster_Internet-e1346361972417.jpg

74. Discussion:
What is a
good…
Time
benchmark
for a build?
Amount of
coverage?
Image: http://www.mdr.de/sachsen/bild235170_v-standardBig_zc-3ad1f7a1.jpg?version=49443

75. Image: http://1.bp.blogspot.com/-pKB3xu6zOc8/TWU1W_4OEqI/AAAAAAAAAzk/ygEOBNumv-Q/s400/nespresso-clooney.jpg
No Waiting…

76. 100%*
Image: http://shirtoid.com/wp-content/uploads/2013/02/100-percent-sci-fi1.jpg

77. Common “Rage” Symptoms
 Slow Automated Tests
 Brittle
 Unreliable
 Large datasets
 Unnecessary integrations
 Inadequate environments
 Too many / UI / manual tests
 Time bound data
 External / Prod integrations
 Reliance on UI / sequence
 False positives / confidence
 Failures being ignored
 Workarounds investigated

78. HEALTH
PROJECT
DEVELOPMENT
TESTING
USER
0
1
2
3
4
5
0
1
2
3
4
5
6
7
1 2 3 4 5 6 7 8
NewRisks&IssuesRaised
NumberofRisks&Issues
Iteration
TEST
COVERAGE
0
5
10
15
20
25
0
1
2
3
4
5
6
7
8
9
1 2 3 4 5 6 7 8
NumberofTests
NumberofDefects
Iteration
MAINTAINABILITY PERFORMANCE
0
200
400
600
800
1000
1200
1400
1600
1800
4700
4800
4900
5000
5100
5200
5300
5400
5500
5600
5700
1 2 3 4 5 6 7 8
LinesofTestCode
LinesofCode
Iteration
OVERALL
$-
$5
$10
$15
$20
$25
$30
0
1
2
3
4
5
6
7
8
9
1 2 3 4 5 6 7 8
BusinessValue
NumberofFeatures
Iteration
Approach: BVC

79. Approach:
Cloud /
Central /
Open
Source ALM

80. “Rage” Lessons Learned
 Treat automated tests with
the same importance as
production code
 Review, refactor, improve ...
 Apply a “Stop the line”
approach to test failure
 Eliminate (quarantine)
unreliable tests
Image: http://us.123rf.com/400wm/400/400/edma/edma0907/edma090700217/5226661-businessman-doing-yoga-exercises-in-a-city-park.jpg
 Ensure collaboration with
developers

81. Greed
Trying to cut costs through automation…
Image: © 20th Century Fox http://blogs.telegraph.co.uk/news/files/2010/09/gordon-gekko.jpg

82. Lure of saving labour
Image: http://www.caradvice.com.au/thumb/640/556/wp-content/uploads/2010/11/Car-salesman.jpg

83. Automation is not cheap!
Image: http://designbuildsource.com.au/wp-content/uploads/2012/03/australian-money.jpg

84. Common “Greed” Symptoms
 Investment in commercial
tools using a business-case
based on reducing headcount
 Using a predicted ROI as a
way of reducing budget for
testing
 Consolidating automated
testing within a special group
Image: © Gracie Films / 20th Century Fox Television http://thegreeneconomy.com/wp-content/uploads/2012/05/greedy-246x220.gif

85. New Software
Test Engineer
Role
New Graduates,
Recruits, Consultants
& Partners
BA
SME
Other
Up
Skill
Assessment
Current Test Analysts
Capable + Current Skills +
Desire
Capable + Desire
Not Capable + No Desire
Not Capable + Desire
**Assessment = Technical Test + Interview**
Approach:
Assist & Assess

86. Approach:
Outsource
Teams Not
Testing
Image: https://sphotos-b-ord.xx.fbcdn.net/hphotos-ash3/p480x480/942876_505183632888249_1411782978_n.jpg

87. Approach: Avoid Test
Automation Projects
Image: http://www.hifi-writer.com/he/progscan/Files/abcestablish.jpg

88. “Greed” Lessons Learned
 Ensure the reasons for
automation are clear
and are NOT based
purely on saving
money/headcount
 Ensure business case
for automation
includes costs for
ongoing maintenance
Image: http://resources3.news.com.au/images/2010/12/02/1225964/296651-piggy-bank-and-coins-chart.jpg

89. Quality
is Key
Image: http://gs-press.com.au/images/news_articles/cache/key_hand_resized-600x0.jpg

90. Approach:
Quality
Assessment

91.  Tests are NOT executed on a regular basis
 State of current defects is unknown
 Functional testing occurs regularly
 All testers can execute the functional tests
 Production verification testing is used to ensure success
 A majority of functional tests have been automated
 Exploratory testing forms part of test execution
 Customers verify implemented features prior to deployment
 A majority of non-functional tests (performance, reliability, ...) are
completed prior to deployment, many are automated
 Testers pair with developers to automate tests
 Customers verify implemented features as they are completed
 Functional and non-functional testing occurs continuously within
development iterations
 Developers and testers are performing test first practices
-1
1
2
3
4
Discussion: Test Execution

92. Wrap
Up
Image: http://www.pria.com.au/sb_cache/priablog/id/924/f/7%20deadly%20sins.jpg

93. Envy Flawed comparison of manual testing & automation
Gluttony Over indulging on commercial test tools
Lust User interface forms the basis for all testing
Pride Too proud to collaborate when creating tests
Sloth Too lazy to maintain automated tests
Rage Frustration with slow, brittle or unreliable tests
Greed Trying to cut costs through automation
7 Deadly Sins

94. Why Use Automation?

95. Image: © Fuzzy Door Productions / 20th Century Fox Television
Don’t Lose
Sight of the
Goal

96. How geeks really workHow Geeks Can Work Together

97. Questions?
Craig Smith
http://www.craigsmith.id.au
craig@craigsmith.id.au
@smithcdau