Matt Robbins Presentation on "Zero to Tested with Docker and Ruby"

1. Zero to Tested
“Bullet proof test automation in hostile
environments.”

2. The test automation promised land
Small team of full stack developers (with specialists in specific areas)
BDD/TDD
Strong adherence to the Test Pyramid
Isolation of different layers (good use of stubs, mocks, fakes)
Thin layer of e2e tests

3. For most projects
Fairly unrealistic, at least not anytime soon...

4. But this means...
“Often reliant on end to end automated tests
which have a terrible reputation!”

5. Why?
People acknowledge e2e tests are essential
Tests are flaky
CI infrastructure is poorly managed (configuration drift)
Development of tests lags behind the development cycle (“test sprints”)
Often ignored by most of the team

6. Worth reading...
http://googletesting.blogspot.co.uk/2015/04/just-say-no-to-more-end-to-end-tests.html
http://www.alwaysagileconsulting.com/articles/end-to-end-testing-considered-harmful/
http://devblog.songkick.com/2012/07/16/from-15-hours-to-15-seconds-reducing-a-crushing-build-time/

7. But I prefer
https://www.symphonious.net/2015/04/30/making-end-to-end-tests-work/
10,000 end to end acceptance tests, primary form of quality control
complete in 50 minutes, including deploying/starting and tearing down the servers
all required to pass before we consider a version releasable
included in our information radiators to ensure the team has constant visibility
are owned by the whole team

8. My theory.
Us “test automation” folk have let ourselves down
Even if we are not in an agile utopia…
Have we exhausted every avenue to make tests reliable?
Need to prove we are doing everything we can to write bulletproof tests

9. Now for the practical stuff!
Even with minimal influence on budget/project direction we can usually hugely
improve on:
CI Infrastructure automation
Automation code and architecture
Let’s take a look at a thin vertical slice through a solid automation setup...

10. CI Infrastructure automation

11. Infrastructure automation
Start from an empty EC2 account and finish with:
Fully functioning Jenkins instance
Automated tests running successfully against a “flaky” test application
Deployed in < 5 mins, very little code, fully automated and “Ephemeral”

12. DevOps Toys - Chef, Packer, Terraform, AWS, Docker
BAKE SPIN UP RUN TESTS

13. Chef

14. Packer

15. DevOps Toys - Chef, Packer, Terraform, AWS, Docker
BAKE SPIN UP RUN TESTS

16. Terraform

17. DevOps Toys - Chef, Packer, Terraform, AWS, Docker
BAKE SPIN UP RUN TESTS

18. Jenkins Job
DSL Plugin

19. Docker

20. Docker

21. Has it finished?

22. Recap...
Automated provisioning of AWS infrastructure
Automated build of ‘pre-baked’ AMI
Automatically provisioned Jenkins with Docker
Docker allows clients to have complete control over dependencies
Job DSL for building jobs

23. Take away
Everything is in source control, no more snowflake CI boxes, no more
handcrafted Jenkins jobs.
“But it’s not scalable!” - switch to agent model with auto scaled agents
“But it’s not secure!”- not at the moment, but easy to add e.g. github auth
More importantly It’s ephemeral - you can destroy if compromised

24. Automation Code & Architecture

25. Tooling choice
In Ruby Land the “go-to” libraries are:
Capybara
Selenium Webdriver
Poltergeist/PhantomJS
SitePrism

26. Automation Code
What are the biggest failings of most end to end regression test suites?

27. My thoughts
Inappropriate synchronisation strategies (page load, async JS)
Inability to deal with non-deterministic applications and environments
Inability to deal with “eventual consistency”
Poor reporting of failure when it occurs

28. RETRY ALL THE THINGS!

29. A flaky app...

30. Page Load
Handled by Selenium Webdriver!

31. Async JS
Avoid this...

32. Async JS
Good libraries have first class support for this:

33. Visibility
Elements exist in DOM but CSS means they are not visible.
Good libraries have first class support for this:

34. Visibility
Worth remembering…
● display:none
● visibility:hidden
● Opacity:0
● Dimensions
And remember there is a W3C spec for Webdriver:
https://w3c.github.io/webdriver/webdriver-spec.html#dfn-element-displayed

35. Non determinism
Should we be expected to deal with this?
Maybe, maybe not
Either way don’t just give up, at least detect, log and report on it!

36. Non determinism
Don’t just rely on re-run - be proactive and report on failure
Libraries probably won’t help here!

37. Eventual Consistency
Caching, load balancing, asynchronous architectures etc.
TL;DR - You might need to wait a bit...

38. Eventual Consistency
You might have to ‘roll your own’ wait helpers from time to time

39. Logging
Configurable levels: debug, info, warn, error etc
Pretty colours - makes debugging broken builds easier
Have log aggregation for your application? (RSyslog, Logstash, Kibana)
Now you can combine test and application logs for super debugging power!

40. Test the tests
We should unit test any “library code” that has significant complexity.

41. Static Analysis
Helps maintain consistency, best practice and readability of code.

42. Reporting

43. Reporting

44. Wrapping up
We need to up our game when writing end to end tests
Make the provision of CI infrastructure automated and ephemeral
Write automation code that can handle hostile conditions
Report on your findings to effect change

45. The code...
https://github.com/mcrmfc/zero-to-tested

46. Questions?