This document provides lessons learned about end-to-end testing based on metaphors from zombie movies. It discusses how end-to-end tests should expect failures like distributed systems due to real-world chaos. Tests need to balance being performant, accurate, repeatable, and meaningful. It provides tips for testing such as using dependency management, controlling all data, avoiding slow test methods, and sharing open-source testing tools and techniques with the community.
Lessons Learned in Test Automation From Zombieland
1. DRAFT
Lessons learned on end to end testing from
Matt Barbour
Sr. Director Engineering
Comcast Silicon Valley
matt_barbour@cable.comcast.com
STAREAST 2016 – May 4th, 2016
2. Confidential and Proprietary
What can we LEARN from a
ZOMBIE MOVIE?
• Characters in zombieland
live by a set of rules in order
to survive the chaos
• Many of these apply in non-
zombie situations too!
Specifically:
End-to-end testing in real
world chaos.
FOL 2014 Process Improvement Project – IDD Process1
Confidential and Proprietary
3. Confidential and Proprietary
End To End Testing as compared to distributed computing
• Distributed systems expect failure
because the real-world is chaotic
• Strive to be fault tolerant by
optimizing for desired
characteristics of the end
application according to the CAP
Theorum
2
• End to end automated tests should
have similar expectations about the
environment they are operating in
Automated
Tests
Performant
Meaningful
Repeatable
Accurate
PARM THEORY
Tests must balance being:
Performant, Accurate, Repeatable, and Meaningful
4. Confidential and Proprietary
KNOW YOUR WAY OUT
BE FORWARD LOOKING AND
OWN THE PROBLEM!!
• Aim for excellence – don’t
tolerate mediocrity
• “that’s the way we have
always done it”™
• Low pass rate
• Tests cant finish quickly enough to
give feedback on a checkin
• Don’t know what a test is doing or
why it exists
• Inconsistent results
• Aim for being in the center of the
PARM matrix
3
Automa
ted
Tests
P
M
R
A
5. Confidential and Proprietary
BUDDY SYSTEM
LEARN AND APPLY DEVELOPMENT BEST
PRACTICES
Developing E2E tests should be no different than developing an application
GTAC 20154
• Design Patterns
• Code Review
• Unit Tests
• Static Code Analysis
Automat
ed
Tests
P
M
R
A
6. Confidential and Proprietary
RULE #31
CHECK THE BACK SEAT
MAKE SURE YOURE NOT BRINGING ALONG UNWANTED DEPENDENCIES
• Use a dependency management tool!
• Explicitly declare all dependencies
• Beware of depending on uber jars in a project…
5
Automat
ed
Tests
P
M
R
A
7. Confidential and Proprietary
A.B.T. ALWAYS BE TESTING
Trim the fat
Look for opportunities to
obtain useful test metrics
outside of a traditional test
Even idle devices can tell you
something!
6
Automat
ed
Tests
P
M
R
A
8. Confidential and Proprietary
MAKE SURE YOUR TEST TOOLS
SCALE
• Use a test framework
which allow you to scale
in both directions
(more devices in parallel)
(more tests in parallel)
• Use open source tools
where possible
• WebDriver isnt just for
web and mobile!!
7
Automat
ed
Tests
P
M
R
A
9. Confidential and Proprietary
TRYING TO DO TOO MUCH IN A
TEST IS A RECIPE FOR FAILURE!
• Tests should focus on one
thing, and one thing only.
• BDD is helpful here
• BDD scenarios should favor
DAMP over DRY
• DAMP = Descriptive and
Meaningful Phrases
• DRY = Don’t repeat
yourself
8
Automated
Tests
P
M
R
A
10. Confidential and Proprietary
Choose your test methodologies
wisely
• You can lose the war against
unreliable, slow tests before you even
start by your choice of test
methodology
• Known offenders
– Image Comparison
– Xpath
– Record/Playback
– Not controlling your data
– The list goes on and on
9
RULE #7
TRAVEL LIGHT
Automated
Tests
P
M
R
A
11. Confidential and Proprietary
RULE #5
NO
ATTACHMENTS
CONTROL ALL THE DATA
IN YOUR SYSTEM
• If you have a data dependent test where you are not controlling the data you
probably have an unstable test, or worse…
• Mocking production data is time consuming… so is triaging false failures
• Tests must be able to start from a known state
1
0
Automated
Tests
P
M
R
A
12. Confidential and Proprietary
WORK AROUND OR ELIMINATE ARCHITECTURAL CONGESTION
11
•Shared resources are the enemy of test
parallelism.
•Your system’s architecture can get in the
way here… don’t assume the application
is infallible. Either work with it or modify
it
•Cloud deployments can mitigate
architectural problems
•VMs are cheap and easy to spin up and
tear down
Automated
Tests
P
M
R
A
13. Confidential and Proprietary
Avoid Strangers
This one is just plain wrong
• The only way we can survive the slow unreliable test zombie apocalypse is by sharing
tools and techniques.
12
All the test tools used by Comcast
Silicon Valley are open-sourced:
https://github.com/Comcast
• Cucumber testing in parallel
• Zucchini
• Device Management
• DAWG
• Selenium Driver Management
• Magic Wand
• Theres a great deal of knowledge
in the community out there, use
it!!
Chaos of zombie world vs chaos of real world
Describe the chaos we are dealing with
Service oriented architecture
One application which runs server side but communicates with hardware in customers homes as well as 5-10 webservices managed by both internal and external groups and released on whatever intervals make sense for that team
15+ different models of hardware and growing
The application itself running in a HA cluster across multiple data centers
Cable TV – no tolerance for downtime
CAP Theorem (Consistent, Available, Partition Tolerant - pick 2)
Lots of analogies between distributed computing (DevOps) and automated testing
PARM theory (Performant, Accurate, Repeatable/Precise, Meaningful - must have all 4)
E2E testing should really be the smallest part of your “test pyramid”
First step is to admit you have a problem
Don’t tolerate mediocrity
You want to have an engineering team which is empowered to solve problems and consider alternative solutions
It’s a fine line – new and shiny doesn’t always mean good for your product
Optimze for being in the middle of the PARM matrix
- For automated tests to be really useful they need to be MORE reliable than the code it is testing
- Analogy to semiconductor measurement
Test Development and application development should be considered one and the same
Treat test authoring as a first class citizen and apply industry standards / best practices
This is in line with treating test development as a first class citizen, but is important enough that it deserves its own slide
Use a dependency management tool – one exists for every language
Lock down your dependencies
Avoid transitive dependencies
Dependency ranges
Maven example
Parent POM for dep mgmt tools which allow inheritance
Uber Jars – leading cause of hair loss among test automation engineers
Trim the fat by :
removing hardcoded sleeps and unnecessary synchronization
Bunch tests which have similar setup steps
Pipeline and parallelize where it makes sense (more on that next)
Look for other ways to get test coverage without adding additional tests
Memory consumption, concurrency, etc
Compare to idle devices
Testing in real world conditions is going to be slower and more problematic than you would like ideally
Combat this by scaling horizontally and vertically and avoiding proprietary frameworks where you can
Scaling horizontal will allow you to get coverage of all your device compatibility tests
- make sure you account for tests which cant run on certain device types
Run same test on multiple instances of the same make/model/browser/os/etc
- will help you suss out inconsitent tests
Just scaling horizontally will still mean you are limited by slow tests. One way to combat this is to scale vertically. It can also be combined with horizontal scaling.
For ex: Run your slow tests in parallel on one set of devices and simultaneously run your other tests in parallel on a separate set of devices
Open source tools (especially the popular ones) have a community around them and are easy to hire people for
Even if your application isnt web or mobile, you can make existing tools work. Things like WebDriver can be adapted to run against other device types
Big monolithic tests are less desireable than narrowly focused tests.
BDD is helpful in this regard
Be careful to avoid antipatterns
Too generic
Too specific so that they lose meaning
Trying to do DRY vs DAMP
One case where good development practices don’t apply
How you implement your tests can have a drastic impact on P, A, & R
When you find anti-patterns in your test methodologies, the first step is to know WHY the test is authored that way
When you get to root cause you may find that there is a better way to do it
If theres no better way to do it you have the knowledge to minimize the damage or just kill the test
Image Comparison example
Failure to control your data is quite often the biggest contributor to unstable tests
If you have a data dependent test you quite likely have an unstable test or worse, youre letting defects get through to production
Why? Mocking prod code is HARD and SLOW… so is triaging false failures or
HOPE is NOT a strategy
Sports example:
Live events are unpredictable
Rain delays, overtime, cancellations, etc
Live events generally happen in off hours
Don’t boil the ocean though – look at what your tests require and mock out only that
Identify architectural congestion in your application or tests
Shared resources are the enemy of test parallelism
Singletons are a big problem here (but not the only one)
First step is to define whether the arch congestion is the correct design for your users. If its not, congrats, you’ve found a defect.
If it is, then you have to figure out how to work around the congestion
Sports example again
Data layer serves multiple clients. If your tests are controlling the data like they should then you will start clobbering yourself once you strat trying to scale our horizontally
Cloud deployments can help here – the brute force approach is to throw up identical clusters.
In the absence of that (either by necessity or choice) you are left with trying to synchronize across tests