1. Effective Test Automation
in an Agile Environment
Krishantha Samaraweera
Senior Technical Lead – Test Automation
WSO2

2. Agenda
● Automated testing
● Success stories
● Automation Basics – Test Pyramid
● WSO2 Test Automation Framework
● WSO2 Platform Automated Test Suite
● Lesson learned
● Automation Basics - Best Practices
● Continuous integration and Continuous Delivery

3. What is Test Automation
● Automation testing – Testing which can be done programmatically
● Compare result – Actual and Expected
● Generate reports
● Goal is to reduce the number of test cases to run manually
● Not to eliminate Manual testing all together

4. Why Automated Testing
● To get early and instance feedback
● Safety net to save time and which is reinvented in to manual
testing.
● Doesn't require human intervention.
● Unattended (Overnight)
● Manual testing has high risk of mission out on something

5. Success Stories
● Google runs 100M+ automated test cases every day
● Google customer surveys team can deliver a change to
production 8 mins after code is committed.
● Ebay runs 35000 test cases in 5 hours
● At Facebook, each of 5,000 engineers commits to trunk HEAD
at least once a day and the code at trunk HEAD is pushed to
production once daily.
● Atlassian JIRA is protected by over 14,000 unit tests and
19,000 functional tests from regressions

6. Success Stories Cont'd..
● Etsy used to take 14 hours and an army of people to deploy
one release into production before CD kicked in. Now it takes
15 mins, 1 person. They did 729 deployments to production in
Nov 2010, only 6 deployment related incidents.
● At Amazon, new code is deployed to production at a
staggering rate of once every 11.6 seconds during a normal
business day.
● That’s 3,000 production deployments per day.

7. Test Automaton Pyramid

8. Test Automation Pyramid Cont'd..

9. WSO2 Test Automation Framework
● Why our own framework
– Open source test scripts
– Stay lean
– Independent
– Single framework for all products
– Facilitate backend testing + UI tests

10. Technology outlineTechnology outline

11. Test Automation Framework Architecture

12. Glimpse of Test Automation Framework
● Simple automation context API.
● Platform wide test execution support.
● Annotation based test execution management support.
● Test reports and coverage report generation.
● Test tooling integration (Jmeter).
● Selenium WebDriver integration and ability to run UI tests on
cross browser environments.
● Framework extensibility capability through pluggable modules.
● Easy retrieval of new configurations.

13. WSO2 Platform Automate Test Suite
● Package tests for other people to use
● Distribution contain all product integration tests and platform
tests jars.
● Test script management tool
● Ant based test executor
● Test execution on distributed product clustered or cloud
deployment.
● Cross platform test execution

14. Lesson Learned
● Once you abandon the maintenance of the automation, it is
likely to die. For a better chance of success, choose an
automation approach that will require the least maintenance.
● Automation requiring multiple cycles and project releases for it
to become fully effective and provide an acceptable ROI.
● Start by automating the most valuable tests.

15. Lesson Learned Cont'd
● Plan your test automation at the beginning of the project.
● Unit Testing FIRST – Fix your test Pyramid.
● Staff should know how to diagnose failures. Minimize calling
automators for every issue

16. Lesson Learned Cont'd..
● Automation development requires the same discipline as
software development.
● Without the cooperation of developers, test automation will be
more difficult than it needs to be.
● Everyone should know what is automated.

17. Test Automation Benefits – Efficiency ROI
Time savings of test execution with automation vs test
execution without automation
Product # of Tests Manual time
per test
case
Iterations Total time
of manual
test
execution
Saved time
ESB 922 8 min 5 384 hours 77 man days
G-Reg 1557 8 min 5 1038 hours 130 man days

18. Continuous Integration
● Continuous Integration is a software development practice
● Team integrate their work frequently
● Each integration is verified by automated build
● Rapid feedback
● Ships better code faster

19. Continuous Integration Cont'd..
Source : http://mikeciblogs.wordpress.com/2010/06/15/continuous-integration-for-agile-project-managers-part-3/

20. Continuous Delivery
How long would it take your organization to deploy a change [to
production] that involves just one single line of code? Do you do
this on a repeatable, reliable basis?”
- Mary and Tom Poppendieck,
Implementing Lean Software Development

21. Continuous Delivery
Build software in such a way that it ready to be released at any
time
– Continuous feedback
– Reduce risk of failures in production system
– Rapid, reliable and repeatable processes – automation!!

22. Test Cases to be Automated
● HIGH RISK – Business critical test cases.
● Features trend to break frequently
● Repetitive tasks are good candidate for automation
● Test cases that are tedious or difficult to perform manually
● Test cases which are time consuming.
● Tests that use multiple data values for the same actions
● Smoke Tests

23. Test Cases not suitable for Automation
● Test cases that are newly designed – Not executed manually
at least once.
● Test cases for which the requirements are changing frequently.
● Usability testing – “How easy is the application to use?”
● One-time testing

24. Testing Philosophy
● You don't have to write 1,000 tests
● The benefit of the tests scales very nicely.
– 1,000 tests is good
– 100 tests is also good
– 10 tests is also good
● And even one test is better than no tests at all

25. Testing Philosophy
● So don't start out by saying "Oh, I have to write 1,000 tests for
this."
● Then you might not write any at all.
● Start by saying "I'll write one test".
● Then write it
● Then maybe write another one
● You will immediately see the benefit of having one test

26. Conclusion
Importance players of successful test automation are right
selection of tools, testing process and team.
Writing tests will make you happy. If you make it into a big
routing task, you won't do it. So don't make it into a big routine
task.
Effective tests can be extremely simple.

27. References
[1]“GTAC 2013: Presentations - Google Test Automation Conference — Google Developers.” [Online]. Available:
https://developers.google.com/google-test-automation-conference/2013/presentations. [Accessed: 17-Mar-2014].
[2]“Continuous Integration for Agile Project Managers (Part 3) | Mike CI blogs.” [Online]. Available:
http://mikeciblogs.wordpress.com/2010/06/15/continuous-integration-for-agile-project-managers-part-3/. [Accessed:
17-Mar-2014].
[3]“All About Automated Testing.” [Online]. Available: http://www.guru99.com/automation-testing.html. [Accessed:
17-Mar-2014].
[4]“Writing Automated Tests.” [Online]. Available: http://perl.plover.com/yak/testing/samples/. [Accessed: 17-Mar-2014].
[5]“Continuous Integration.” [Online]. Available: http://martinfowler.com/articles/continuousIntegration.html. [Accessed:
17-Mar-2014].

28. Thank You