We've come to realize that automation provides an immense amount of value in preventing regressions and helping to deliver quality software. As your automation grows and grows, it requires continuous maintenance so that tests remain fast, reliable, and valuable. If you're not scaling efficiently, your automation suite will turn into a messy, uncontrollable beast. Having a lean test suite will help to combat this. This presentation shares methods that you can use to keep your automated test suites lean and mean, so they always provide quick and accurate feedback to your software delivery team.

1. KEEPING
YOUR TESTS
LEAN

2. @iammeaghanlewis
meaghan@lever.co
meaghan-lewis

3. WHAT DOES
LEAN
MEAN ANYWAY?

4. VALUABLE
RELIABLE
FAST

5. 1000+ TESTS

6. 12 HR BUILD

7. 100% FAILURE RATE

10. HOW DID WE GET
A LEAN
TEST SUITE?

12. NOT QUITE...

14. VALUABLE

15. Tests provide useful
information to your team

16. TESTS CAN GET OUTDATED

17. Ideas for VALUABLE tests

18. Re-evaluate & Analyze
● What is this test doing?
● What value is this test providing?
● Is this testing the right thing?

19. Quality vs Quantity

20. Collaborate on testing

21. Treat test code like
production code

22. RELIABLE

23. Tests should always have
the same result

24. Antipattern: Unreliable tests
EVEN ONE FAILURE MAKES A DIFFERENCE...

25. Ideas for RELIABLE tests

26. Isolate flaky tests
Isolate
flaky test
Analyze Fix
Move
back to
build

27. Monitor flakiness
● For each test run, record the failures
○ Test
○ Failure
○ # of times failed

28. Automatically rerun failures

29. Use specific test selectors

30. Independent tests
● Tests should all be hermetic
● Execution of one test does not affect others
● Run in dedicated testing environment

31. FAST

32. TESTS HAVE A SHORT FEEDBACK
LOOP

33. TESTS SHOULDN’T FEEL LIKE A MARATHON

34. UNIT &
INTEGRATION
TESTS
DEPLOY TO
STAGING UI TESTS DEPLOY TO
PRODUCTION
POST DEPLOY
TESTS
CONTINUOUS DEPLOYMENT PIPELINE

35. Ideas for fasts tests

36. Parallelize

37. Avoid testing in the UI

38. Headless tests
● XVFB ● PhantomJS

39. REMEMBER...

40. There are tradeoffs

41. It’s okay to start over

42. Testing is a team effort

43. THANKS!
@iammeaghanlewis
linkedin.com/in/meaghan-lewis/
meaghan@lever.co