Automation ROI Microsoft-E4D Event

1. Automation
Testing
ROI
Presented by:
Yaniv Cherkasky
yaniv@e4d.co.il
054-7722568

2. Agenda
• First Session
– Automation – Why? When? When Not?
– Test Automation Frameworks Models
– Guidelines for first approaching Automation
• Second Session
– Automation ROI
– Improving ROI
– Case Study

3. Automation Testing Definition
Test automation is the use of software to
control the execution of tests, Commonly, test
automation involves automating a manual
process already in place that uses a formalized
testing process. (Wikipedia)

4. When?
• Repeatable Test Scenarios
• Application Logic Level
• Unit Test the code
• Checking the Performance
• Data/Event Driven Tests
• Multiple configurations

5. When Not?
• Test Cases that are complicated for coding
(e.g. Image compression/Video quality etc.)
• Test Cases that we run only a limited number
of times
• Premature parts in the Application (Constant
Changes)

6. Why not Automate Everything?
• Low ROI (Return On Investment)
• Many times the Test Case is too complicated
for a machine
• Most of the time the Look and Feel needs to
be done by Human Tester

7. Why do it?
• Length of the testing phase is reduced
• Higher efficiency (Save Time)
• Improve Productivity (Increase bugs coverage)
• Can reduce testing manpower (save money)
TTM
Resources Quality

8. Test Automation Types
Load and Performance testing
Installation and configuration testing
Boundary testing
Unit Testing
Compliance testing
Smoke and unit testing
Localization testing
Regression testing

9. Test Automation Framework
common Models
• Data Driven Automation Framework Model
– Mass storage of data
– Limited to the automation tool in use
– Easy to implement, hard to maintain
• Keyword Driven Automation Framework Model
– Data tables using a keyword vocabulary
– Hard to implement, easy to maintain

10. Test Automation Framework
common Models
• Test-driven development (TDD)
—Focus on the task at hand
—TDD leads to more modularized, flexible, and
extensible code
—Create unit tests before writing the code

11. Automation Entry criteria
Frequent
builds/patches/iterations
Test cases that
need verification
with a high
precision
Test case
functionality with
high complexity

12. Automation Exit criteria
Successful
Execution
Script has been
reviewed for
coding standards
Review
completed for
variables,
functions, test
steps etc.
Report Version
Repository controlled

13. Guidelines for first approaching
Automation
1. Perform full analysis breakdown of system features & modules
relationship
2. Perform full analysis for manual test cases
3. Mark the test cases which are repeatable which has long
execute time
4. Do a research of the environment (technology used, SW/HW
limitations, platforms etc.)
5. Make a check list for the required testing types (Functionality,
GUI, Performance etc.)
6. Pickup the appropriate test automation tool (MTM )

14. Guidelines for first approaching
Automation
7. Pick the suitable framework structure
8. Conduct a breakdown of all framework methods
9. Mark the common use of framework methods in all Test Cases
– these methods will become your first building blocks!
10. Calculate ROI
11. Prepare a presentation with Automation Framework structure
& ROI calculation
12. Pick an example
13. Run a POC

15. Tips & Tricks
• Make sure your are considering future maintenance in your
planning – maintenance cost money!
• Don’t start with demanding resources, start with 1 resource for
POC
• Use also authentic factors for the ROI (items related to the
unique product/environment etc.)
• Automation is a development group, manage it as one!
• When you approach your boss, in any given time, make sure
you are translating all to budget terms :-)
• Monitor Automation ROI on a yearly basis comparison

16. Questions?

17. Automation ROI
It is important to consider the following factors when evaluating
the ROI of automation:
• The cost of executing the automated tests. Consider the
infrastructure required
• The cost of maintaining the automation as the product evolves
release by release.
• The cost of any additional tasks necessitated by the automation
• ROI will vary depending on your business environment and the
application under test.

18. Importance of ROI
• ROI is as important if not more important than technical
automation topics
• Test Automation is an investment

19. Return On Investment
TTM
Resources Quality

20. ROI Challenge
UNREALISTIC EXPECTATIONS
• 100% Automation
• Immediate Payback
• No Automation
Analysis

21. ROI Dilemma
• Continued automation justification
• Difficult to meet unrealistic expectations
True test automation benefits:
Increased Cost Savings
Increased Efficiency
Increased Quality

22. Calculating ROI - Factors
Automation Testing Factors
– Tool Costs
Fixed
• Software
• License
– Execution Machines
– # of Tests to Automate/Effort Estimation
Variable
– Automated Development/Debugging Time (average per test)
– Automated Test Execution Time (average per test)
– Automated Test Analysis Time
– Automated Test Maintenance Time

23. Calculating ROI – Factors
Return on Investment
Automation ROI
Number of hourly employees? 500
Average hourly rate of employee? 16.00
Number of minutes to review/total each time card (APA 7)? 3
Hourly rate of person reviewing time card? 16.00
Percentage of inaccurate time cards (APA 1-8)? 2%
Number of standard hours an employee worked in a pay period? 40.00
Daily lost time in minutes for an employee (Calculation, Rouding) (APA 49)? 20
Card Audit Savings
Total savings in minutes 1000
Total minutes saved weekly 5000
Total hours saved weekly 83.3
Savings in dollars weekly $1,333.33
Monthly savings $5,773.33
Yearly Savings $69,333.33
Savings in Lost Time
Daily lost productivity in hours 0.33
Average wages overpaid daily $5.28
Average wages overpaid weekly $26.40
Total wages recaptured weekly $13,200.00
Total wages recaptured monthly $57,156.00
Total wages recaptured yearly $686,400.00
Savings from Eliminating Errors
Total weekly payroll expense $320,000.00
Weekly savings $6,400.00
Monthly savings $27,712.00
Yearly Savings $332,800.00
Total Savings Summary
Weekly audit savings $1,333.33
Weekly error savings $6,400.00
Weekly wages recaptured $13,200.00
Weekly payroll savings $20,933.33
Monthly payroll savings $90,641.33
Yearly Payroll Savings $1,088,533.33

24. Calculating ROI for Automation Projects
is an on-going task…
Let:
NPV(ATCF) = Net Present Value of After Tax Cash Flow generated by the
investment
NPV(IC) = Net Present Value of Required Investment
Profitability Index = 100*NPV(ATCF) NPV(IC)
− NPV(IC)
An outline of the equations underlying this expression is given in the Appendix.
The elapsed time to positive cash flow, i.e. the time at which the
discounted cumulative cash flow crosses into the positive region is also important.
This is sometimes called the discounted payback period.

25. Improving ROI
Improvement Result

26. Improving ROI
Improvement Result

27. Automation ROI
Case Study
On-Line Shop
Order Pages

28. Automation ROI – Case Study
• Order pages test includes verifying the on-line buying of
products
• Effort Estimation for manual testing takes about 2 weeks for 1
person
• The frequency of running the test is quite common since full
regression is needed each time a change is made in results
from a new marketing campaign or even a new discount
• Manual test does not cover all buying aspects (boundary
cases) – but only the positive testing required by the
requirements
• Currently, the test duration harms the project dynamic
proceeding

29. Automation ROI – Case Study cost$ per month
Manual
Testing
(today) Automatic testing
Effort in days $ cost Effort in days $ cost
Sickness 1 days 113.63 0 days 0
Holiday 1.5 days 170.44 0 days 0
Test Development time 7 days 795.41 19.5 days 2215.785
Test Execution time 14 days 1590.82 1.5 days 170.44
Bugs analysis time 5 days 568.15 1 days 113.63
Report creation 3 days 340.89 1 days 113.63
Test maintence time 1 days 113.63 2 days 227.26
Total: 3692.97 Total: 2840.745
Legend:
Month Total $
Tester Hourly Rate 113.63 22 days 2500
Automation Fixed costs
Testing tool 3000

30. Automation ROI – Case Study
Conclusions
• Currently on “Order Pages” tests we save $ 852.225 per
month (for 1 run) it’s $ 10226.7 per year by using
automatic tests (numbers include lost business,
refunds and testing the complex scenario with all it’s
factors manually)
• Currently in 1 year there are about 5 campaigns of to be
tested…doing it with manual tests will take 120 days. If
using automation for it, it will take 7.5 days. It means:
112.5 working days that are saved!!!

31. Automation ROI – Case Study
Conclusions
• Automation tests can run 10 times more in 1 month then
using manual tests which means: more regression =
more stable product
• Automation 1 test run duration = 7 peoples work
• The cost for the long run is actually decreases!
• Most of the automation costs are fixed costs (one-time
expenses) and are for the short run

32. Questions?

33. Thank You!!!