Test faster, release faster, get to market faster. Automated testing is the future of software development but all too often the performance and longevity of the tests themselves are an after thought. This lightning talk discusses why they shouldn't be.

1. Automated)Tes,ng,)Con,nuous)Integra,on)&)Con,nuous)Deployment)
Test)Performance:)What)it)is;)Why)it)ma@ers)

2. What Makes for a Good Automated Test?
• Eliminate repetitive manual tests
• Yield reproducible results
• Cover common and edge/error cases
• Ideally either pure unit or integration test
• Strive for clarity even when code is gnarly
• Test for business requirements

3. Automated Test Suites are Software Artifacts, too
• The Good:
– Catch errors early and often
– Eliminate repetitive manual tasks
– Reduce integration risk
• The Bad:
– Is software too and has all the same problems
• Often become bloated over time
• Rarely documented
• The Ugly:
– Not the primary deliverable and get short shrift
• I.e. the Bad get Worse

4. What is “Test Performance”?
What do we mean by “test performance”:
– How long does the suite take to run?
– Can I run it locally or do I need Big Iron?
– Does the suite check valid, useful assertions?
– How much effort does it take to maintain and update?
To Wit: Is it useful and how many cups of coffee?

5. Technique: Software Engineering
1. Encapsulation and separation of concerns
• A fast test suite that is a nightmare to maintain is
still slow in terms of human time consumed
2. Tackle test performance problems in sprints and
apply Amdahl’s law aggressively
3. Separate suite into unit/integration or small/medium/
large, etc. and run more expensive tests only daily
4. Remove dead code early and often

6. Technique: Software Optimization
Software optimization may be warranted:
– Many test suites are I/O intensive
• Flash and big memory can help
• Often the bottle neck is database tuning
1. Turn off fsync
2. Avoid synchronous DDL updates
3. Use truncation rather than transactions
– Headless browsers such as phantomjs are cheap
and often have sufficient fidelity

7. Technique: Throw Hardware at the Problem
My local development environment:
1. 4 Fast Cores
2. 16GB of SDRAM
3. 256GB of Flash
When a single core (or single node) is not enough, batch
parallelism can take you a very long way
Goal: avoid software optimization; cycles are cheap

8. Technique: Randomization
Randomization is a favorite testing technique:
1. Black-box security testing through random input
generation (aka: “fuzzing”)
2. Draw input sizes or arrival times from a known
distribution for performance testing
3. Shuffle order tests are run in to trip hidden,
unintentional dependencies
Randomization is a good technique when “interesting”
examples are sufficiently common

9. Avoiding Randomization
Real world example: an LZW/LZ77-style compressor
– Used widely in a major commercial storage appliance
– LZ-style algorithms operate by implementing a simple
state machine that is prone to buffer overflows
1. Performance & correctness both hard requirements
2. Performance gated on memory hierarchy
• Tested data-flow for performance before full
correctness regression testing
3. “Interesting” test cases exercised all edges in state
machine diagram
• Distribution matters – choosing inputs uniformly
at random nearly useless