The document discusses a Testing Tools Management (TTM) methodology for managing testing automation projects. It outlines common issues like failing to implement tools properly or meet expectations. The TTM methodology involves: 1. Analyzing systems and recruiting developers and managers before purchasing tools. 2. Calculating ROI by considering factors like product age and releases. 3. Ensuring readiness through stable environments, business process understanding, and tool/environment preparation. 4. Following stages of development like sanity testing, quick ROI packages, modular regression, and data inflation. The methodology also includes defining scope, following up, documenting processes, and producing summary reports. The goal is to properly implement tools and realize expected ROI,

1. TTM methodology
The right way to manage your Testing Automation
projects

2. Testing Automation projects - life cycle
Common life cycle
1 Purchase the tool 2 Recruit developers 3 Understand ROI 4 Fail to implement
TTM – life cycle
Analyze the systems Recruit developers Follow TTM
1 2 3 Understand ROI 4
Purchase the tools and managers methodology

3. Testing Tools Management
Methodology

4. ?Why Do We Fail
We have the best people and the best
tools but still many projects are failing
management expectations

5. What is our management
expectation
•Testing Automation ROI
•Coverage
•Resource saving
•Shortening execution period
•Testing Automation usage

6. How to build Testing Automation ROI
-input parameters
Age of the product
Releases planned for a year
Required regression cycle per year
Number of existing regression test cases
Number of configurations to be tested
Cost of the testing resources
Infrastructure cost
Test automation tool cost
Percentage of test cases that can be considered for automation
Time required for building reusable components, automation
frameworks and batch scripts
Hourly cost per test automation resource

7. Testing Automation ROI – Points of benefit
3

8. Testing Automation ROI – Calculator 3
Microsoft Excel
97-2003 Worksheet

9. Testing Automation Readiness 4
• Developers
• Stable environment
• Understand business process
• Tools readiness
• Testing Automation environment readiness
• Developers understand the project scope
• Set expectation

10. 4
Testing Automation Stages
• Sanity SET
– Set of ~50 QTP scripts, verify that new build (version) ready for massive
testing process
• Quick ROI Developments
– Packages can retrieve the investment immediately (e.g. API, data driven, data
inflation)
• Regression by modules
So important to make sure NOT to develop all regression tests in one SET, “Before
moving forward with the development stage, we need to analyze our system and
divide it by modules, for each module to develop his own automation SET”
• Data Inflation
– Using our GUI interface can help with massive data changes

11. 4
Why Do We Fail con’t
How to succeed with your testing automation project using the TTM
•Understand the application and the
development process
• Management support (not enough)
• High expectations for immediate ROI
– Underestimating the time, cost and effort for the initial introduction
and implementation of a tool (e.g. training, external expertise and
changes to the testing process).
– Underestimating the effort required to maintain the test scripts and tool
updates

12. 4
Testing Tools Management Progress
• Define the project scope
• Follow up
• Testing Automation methodology
• Summary

13. 4
Define the Project Scope
• High Level Design (Test Plan)
– The document contains the Set we are about to
automate, the concept of automation,
verification points, parameters, etc…

14. (Define the Project Scope (Con’t 4
• Design (Test cases QC)
– Test cases developed in QC cover all required area in the
application

15. (Define the Project Scope (Con’t 4
• Estimation module
– A tool can assist to understand how many working days
we need for the project, include all required steps:
– Testing priority
– All Automation steps
– Risk factor

16. (Define the Project Scope (Con’t 4
• Work plan
– Following the estimation module solution the exact
work plan by days

17. 4
Follow Up
• Test Cases list

18. 4
(Follow Up (Con’t
- Work Plan

19. Testing Automation Methodology
• QTP Standard and Guidelines 4
– Documents introduces basic guidelines and code methods
for scripting via QTP. The document describes all
acceptable practices to develop in QTP:

20. Testing Automation Methodology 4
((Con’t
• Code inspection
– All of the inspectors to reach consensus on a work
– Product and approve it for use in the project

21. 4
Execution Daily reports
No. Estimated
No. of
TA SET of Status Executer time to SET Log PATH Defect # Comments
errors/failure
TC’s complete
BI 182 FAIL Dotan 2 WD Link to BI Report Till today covered 50 TC’s
API Interaction 18 3 PASS 3 WH Link to API Interaction report Yesterday Configuration issues
Tali
defects close
API Application 19 1 PASS Irmi 1WH Link to Application API Testing automation errors
API Application 49 1 PASS Irmi 1WH Link to Application API_1
Ticketing 46 2 PASS Dotan 1 WH Link to Ticketing report Analyze by Dotan
EMT 18 2 PASS Dotan 1 WH Link to EMT report
A2A 10 PASS Dotan 30 Min Link to A2A report
Voice 12 1 PASS Tali 3H Link to Voice Report Testing automation errors
Sanity 14 1 PASS Lior 45 Min Link to Sanity report
JT 17 8 FAIL 1 WH Link to JT Report Analyzed by Dotan
Tali
Link to JT Report_1
Reports 53 17 PASS Tali 1 WH Link to Reports log Data issues need to be fixed!!!

22. Execution Daily reports
# of
Module written Testing coverage # of pass TC's # of fail TC's FAIL TC's
TC's
# % # % # % 200
BI 182 20 11 10 50 10 50
API Interaction 18 18 100 18 100 0 0 180
API Application 68 68 100 66 97 2 3 160
Ticketing 46 46 100 40 87 6 13
EMT 28 18 64 15 83 13 72 140
A2A 10 10 100 10 100 0 0 120
Voice 11 11 100 11 100 0 0 # of written TC's
Sanity 14 14 100 12 86 2 14 100
C
T
o
# of fail TC's
s
#
JT 17 17 100 17 100 0 0
f
'
80
Report 53 53 100 40 75 13 25
60
40
Coverage BI
20 API Interaction
200 0
API Application
Ticketing
180 EMT
160
A2A
Voice
140 Sanity
JT
120
Report
# of wri tten TC's
100
C
T
o
Testing coverage
s
#
PASS TC's
f
'
80
FAIL
60 200 BI
API Interaction
40 180 API Application
20 Ticketing
160
EMT
0 140 A2A
Voice
120
Sanity
# of written TC's
100 JT
C
T
o
# of pass TC 's Report
s
#
f
'
80
60
40
20
0

23. Summary Development Stage
4 Follow TTM
• Final Report methodology
• A document contains all project steps include:
• Test case list
• Testing automation coverage
• Functions
• Folder where script
• How to execute the SET
• Result example

24. (Summary Development Stage (Con’t m
• Code Freeze
– A process to save all developed code under one place.
(With a limited access)

25. !!!Thank you All
?Any Questions