The document discusses quality assurance (QA) metrics in agile development. It begins by defining quality for both products and processes, noting that QA influence increases as development moves from requirements to validation. It then covers the types of metrics that can be used as a foundation for measuring product quality, including quantitative, qualitative, absolute, relative, and derivative metrics. Finally, it provides examples of QA metrics that can be used for daily monitoring of quality, as well as metrics that can be included in regular quality reports for sprints and releases.
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
Â
QA metrics in Agile (GUIDE)
1. QA Metrics in Agile
Look at the Quality from different angels. Real appliance.
By Volodymyr Prymakov
www.qaexperts.pro
2. About Me
Volodymyr Prymakov, Ukraine
Head of Performance QA Unit at TCoE,
Interim Head of TCoE at Thomas Cook,
Co-owner at QAExpert.pro
â˘14.5 years in QA
⢠>45 projects experience
⢠Certified: ISTQB Advanced Test Manager/Test
analyst, ICAgile CP, SAFE, CSPO
/in/vladimirprimakov/
www.qaexperts.pro
5. Presentation Plan
1. What is quality of product and process
2. What is foundation for metrics
3. Types of Metrics
4. QA Metrics for Daily Monitoring
5. Regular Quality Reports
6. QA trend Metrics
7. User Satisfaction and Qualitative metrics
8. Team Efficiency Metrics
9. Automation Metrics
10. QA Metrics tools
www.qaexperts.pro
7. What is quality of an application
Requirements
fit business
needs
Requirements
Has high
quality
Product
corresponds
to
requirements
Product
No
Outstanding
Defects
Product
satisfies real
life conditions
Product
Meets user
needs
Minimal
QA Influence
Medium
QA Influence
High
QA Influence
High
QA Influence
High
QA Influence
Guessing Verification Verification Verification Validation Validation
Minimal
QA Influence
www.qaexperts.pro
8. What is quality of an application
Requirements
fit business
needs
Requirements
Has high
quality
Product
corresponds
to
requirements
Product
No
Outstanding
Defects
Product
satisfies real
life conditions
Product
Meets user
needs
Minimal
QA Influence
Medium
QA Influence
High
QA Influence
High
QA Influence
High
QA Influence
Guessing Verification Verification Verification Validation Validation
Minimal
QA Influence
www.qaexperts.pro
Deviation in Quality
9. What is quality of an application
Requirements
fit business
needs
Requirements
Has high
quality
Product
corresponds
to
requirements
Product
No
Outstanding
Defects
Product
satisfies real
life conditions
Product
Meets user
needs
Minimal
QA Influence
Medium
QA Influence
High
QA Influence
High
QA Influence
High
QA Influence
Guessing Verification Verification Verification Validation Validation
Minimal
QA Influence
www.qaexperts.pro
Deviation in Quality
10. What is quality of an application
Requirements
fit business
needs
Requirements
Has high
quality
Product
corresponds
to
requirements
Product
No
Outstanding
Defects
Product
satisfies real
life conditions
Product
Meets user
needs
Minimal
QA Influence
Medium
QA Influence
High
QA Influence
High
QA Influence
High
QA Influence
Guessing Verification Verification Verification Validation Validation
Minimal
QA Influence
www.qaexperts.pro
Deviation in Quality
11. What is quality of an application
Requirements
fit business
needs
Requirements
Has high
quality
Product
corresponds
to
requirements
Product
No
Outstanding
Defects
Product
satisfies real
life conditions
Product
Meets user
needs
Minimal
QA Influence
Medium
QA Influence
High
QA Influence
High
QA Influence
High
QA Influence
Guessing Verification Verification Verification Validation Validation
Minimal
QA Influence
www.qaexperts.pro
Deviation in Quality
12. What is quality of an application
Requirements
fit business
needs
Requirements
Has high
quality
Product
corresponds
to
requirements
Product
No
Outstanding
Defects
Product
satisfies real
life conditions
Product
Meets user
needs
Minimal
QA Influence
Medium
QA Influence
High
QA Influence
High
QA Influence
High
QA Influence
Guessing Verification Verification Verification Validation Validation
Minimal
QA Influence
www.qaexperts.pro
Deviation in Quality
13. What is quality of an application
Requirements
fit business
needs
Requirements
Has high
quality
Product
corresponds
to
requirements
Product
No
Outstanding
Defects
Product
satisfies real
life conditions
Product
Meets user
needs
Minimal
QA Influence
Medium
QA Influence
High
QA Influence
High
QA Influence
High
QA Influence
Guessing Verification Verification Verification Validation Validation
Minimal
QA Influence
www.qaexperts.pro
Deviation in Quality
14. What is quality of an application
Requirements
fit business
needs
Requirements
Has high
quality
Product
corresponds
to
requirements
Product
No
Outstanding
Defects
Product
satisfies real
life conditions
Product
Meets user
needs
Minimal
QA Influence
Medium
QA Influence
High
QA Influence
High
QA Influence
High
QA Influence
Guessing Verification Verification Verification Validation Validation
Minimal
QA Influence
www.qaexperts.pro
Bad
Product
Deviation in Quality
17. What is quality of Process
Quality Process brings much Business Value with keeping project or program engine safe and efficient
1. Deliver new functionality often
2. Deliver more functionality
3. Delivery quality software
4. Delivery what is planed
5. Deliver with minimal risks
6. Business trusts dev teams, processes, and produced results
www.qaexperts.pro
1. Project vision, goals, roadmap, and other plans are clear to everybody
2. Process is transparent and strait-forward
3. Product quality is always transparent
4. Technical dept is being eliminated
5. Efficient Communication and Interdependency solving
6. Shared decisions making and responsibility
7. Lessons learned and continuous improvement are in place
8. High team sprit and satisfaction
18. What is quality of Process
Quality Process brings much Business Value with keeping project or program engine safe and efficient
1. Deliver new functionality often
2. Deliver more functionality
3. Delivery quality software
4. Delivery what is planed
5. Deliver with minimal risks
6. Business trusts dev teams, processes, and produced results
www.qaexperts.pro
1. Project vision, goals, roadmap, and other plans are clear to everybody
2. Process is transparent and strait-forward
3. Product quality is always transparent
4. Technical dept is being eliminated
5. Efficient Communication and Interdependency solving
6. Shared decisions making and responsibility
7. Lessons learned and continuous improvement are in place
8. High team sprit and satisfaction
19. What is quality of Process
Quality Process brings much Business Value with keeping project or program engine safe and efficient
1. Deliver new functionality often
2. Deliver more functionality
3. Delivery quality software
4. Delivery what is planed
5. Deliver with minimal risks
6. Business trusts dev teams, processes, and produced results
www.qaexperts.pro
1. Project vision, goals, roadmap, and other plans are clear to everybody
2. Process is transparent and strait-forward
3. Product quality is always transparent
4. Technical dept is being eliminated
5. Efficient Communication and Interdependency solving
6. Shared decisions making and responsibility
7. Lessons learned and continuous improvement are in place
8. High team sprit and satisfaction
23. Definition of Done (DoD) defines
Which quality Product should
have?
Which Way and How
thoroughly a product or
functionality should be
tested?
Min Sufficient Max Allowed
www.qaexperts.pro
24. Definition of Done for Stories, e.g.
⢠Coded and integrated to main branch,
⢠Unit tests are written and passing,
⢠API component tests are written and passing,
⢠Code reviewed and conforms to coding standards
Passed Dev
Testing:
⢠Code deployed to QA environment. Automatic API-integration and
EndToEnd smoke testing tests are passing,
⢠Test Design is completed,
⢠Positive and negative Acceptance Tests are (automated and)
passed on platforms of priority 1. Exploratory tests are passed on
platform of priority 2.
⢠Selective Regression is carried out by a necessity and passed.
Passed QA testing:
No (P1/P2)
bugs
outstanding
for STORY
and neighbor
functionality
Story meets INVEST criteria
Story is Signed-Off by Product Owner
www.qaexperts.pro
25. Definition of Done for Sprint, e.g.
⢠Stories that meet DoD are considered DONE, the rest goes to
next sprint.
⢠It should be avoided to merge Risky or buggy code to main
branch at the end of sprint!
Stories meets DoD
⢠Regression testing is passed with P1 test cases on at least 1
top priority platform/device
⢠Smoke or exploratory testing is passed on platforms/devices
of priority 1 and 2
Cover Regression
Test
⢠Main Integration tests for live 3d-party services are passed on
STAGE or other environments3d party Integration
Test
No (P1/P2)
bugs
outstanding
for product
and neighbor
products.
Sprint is Signed-Off by Chief Product Owner
www.qaexperts.pro
26. Definition of Done for RELEASE, e.g.
⢠All Stories, which were merged to Main branch, meets DoD for stories.
⢠Risky or buggy code is not merged to main branch or is toggled off or is
reverted!
Stories meets DoD
⢠Regression testing is passed for all priority 1 platforms/devices.
⢠Exploratory testing is passed for priority 2 platforms/devices.
⢠Smoke testing is passed for priority 3 platforms.
Full Regression Test
⢠All Integration tests for live 3d-party services are passed on STAGE or
other environments
⢠There is no other 3rd party release ongoing that may impact release
negatively
3d party Integration Test
and Releases
⢠UAT is passed on STAGE environment
UAT
⢠Performance Testing successfully passed
⢠Security Testing successfully passed
⢠and other types of testing and preparations to release
Other Types of Testing
No (P1/P2)
bugs
outstanding
for product
and neighbor
products
Release GO Aproval from by Chief Product Owner, QA team, Release Manager and
respective Business representatives
All other unresolved
bugs are tracked,
reviewed, and planned
when to be fixed
www.qaexperts.pro
30. Types of Metrics
Quantative - Information that is directly measured as a
number
Qualitative - Information that results from human
observation and analysis
www.qaexperts.pro
31. Types of Metrics
1. Absolute metrics
2. Relative metrics
3. Derivative metrics
Purpose of Metrics.
1. Goal Metrics: To measure results/goals
2. Actionable metrics: to make decision
3. Predictive metrics: to predict future
www.qaexperts.pro
32. Stages at which QA and other metrics are needed
Implementation period
⢠Daily Monitoring of
Quality, product
implementation, and
test progress
Key sprint and release
points
⢠QA Report Before
sprint Demo
⢠QA Reports During
stabilization /
regression / UAT
period
⢠QA report Before
Release
⢠QA Reminder report
when product has
critical quality state
or low product
implementation or
testing progress
After Release
⢠Release quality and
process analysis
⢠Quality and process
Trends and analysis
⢠Team efficiency
analysis
www.qaexperts.pro
34. Goals of Daily Quality Monitoring
1. What is the current quality of the
product?
2. How much work is left?
3. Do we have enough time to finish and
stabilize the product?
www.qaexperts.pro
35. !!!Important:
Breakdown Defect Metrics
Always breakdown your metrics to the following groups:
All Bugs! â shows general product quality state
Critical and Blocker bugs â Shows whether we
meet Readiness criteria!
Lower Priority bugsâ Shows the rest of bugs,
general amount of which can negatively influence on the
end-userâs impression about the product.
www.qaexperts.pro
36. Defects reported by Date
Critical and Blocker bugs Major and lower priority bugs
Jira âRecently Created Chartâ Gadget
Allows you track new bugs and immediately react to them!
www.qaexperts.pro
37. Open Defects reported Today
Jira âFilter Resultsâ Gadget
Allows you to quickly review new bugs
www.qaexperts.pro
38. Open Defects By Component/Platform
Major and lower priority bugs
Jira âPie Chartâ Gadget
Critical and Blocker bugs
Shows which
component/platform
is not ready for release
or requires bug fixes
www.qaexperts.pro
39. Defects By Component/Platform for Current
Affected Version (any Status)
Major and lower priority bugs
Jira âPie Chartâ Gadget
Critical and Blocker bugs
Shows which
component/platform
require more thorough
testing,
e.g. (Selective)
Regression testing
www.qaexperts.pro
40. Critical Open Bugs by Status
Jira âIssue Statisticsâ Gadget
Allows you to quickly get Status of Open Critical Bugs and understand whether we have enough time to fix them
www.qaexperts.pro
41. Open Bugs by Priority
Jira âIssue Statisticsâ Gadget
Shows general quality of the product
www.qaexperts.pro
42. !!!NOT-assigned Open Critical Defects
Jira âIssue Statisticsâ Gadget
To Fix Version
To Team
(for big teams only)
To Sprint To Developer
Shows probably lost or forgotten Critical open bugs, which should be processed or escalated immediately!
www.qaexperts.pro
43. www.qaexperts.pro
Workload:
Assigned Open Tasks and Bugs for Developers/QAs
Jira âIssue Statisticsâ Gadget
By teams By Developers
For current Fix Version or sprint
By QAs
Stories
By teams By Developers By QAs
Bugs
To assure that there is no bottlenecks and overloads. To Remind about
work assigned!
To spread load equally between teams, Developers and QAs!
44. Release scope status by developers/QAs/teams
If by the end of sprint or before stabilization period at release a lot of stories are in open status,
it can bring risk of late merge status, not enough time for testing, not ability to finish functionality etc.
Action: Try to spread load equally by teams, QAs and Dev Specialists. Remind about work Not Done.
Postpone certain functionality to next Release!
Stories Bugs
Jira âTwo Dimensional Filter Statisticsâ Gadget
www.qaexperts.pro
45. Stories/Bug Status By Fix Version
created >= -365d
Stories Bugs
created >= -180d
Allows to understand ability to finish/fix all tasks or bugs in time for a certain sprint/release
Jira âTwo Dimensional Filter Statisticsâ Gadget
www.qaexperts.pro
46. Current Fix Version Progress
Shows general release/sprint ProgressJira âRelease Reportâ
www.qaexperts.pro
48. Kanban board for All Open Critical and Blocker Bugs
(For big teams)
Allows to
accumulate
and track all
open Critical
and Blocker
bugs in 1 place
www.qaexperts.pro
49. Regression testing
Recommended Statuses:
⢠Failed with Critical and Blocker bugs
⢠Failed with Major and lower priority bugs
⢠Failed by automated tests
⢠Blocked
⢠Not tested
⢠In Progress
⢠Passed
www.qaexperts.pro
50. Regression Testing Coverage and Results
For Release/Milestone
â General Results
By Cycle/Stages
- Every cycle should has certain purpose, e.g:
1. Initial Feedback about quality,
2. Exploratory testing,
3. Confirmation testing
4. 3rd Party services testing.
www.qaexperts.pro
51. Regression Testing Coverage and Results
By Product Component
⢠by platform/device
⢠by test priority:
⢠Priority 1
⢠Priority 2
⢠Priority 3
By Type of Test:
⢠Smoke test
⢠Regression test,
⢠UAT test
www.qaexperts.pro
54. Sprint Report
1. Story Delivery rate % = Actual Stories Completed vs. Committed Stories, e.g.: 90%
2. Amount of Stories met DoD, e.g.: 25
3. Amount of Stories Moved to a new sprint, e.g.: 4
Because of 1 not implemented, 1 Reopen because of Critical Bugs, 2 Not Tested
Specifies the reasons, risks and recommendations.
4. Regression testing coverage and results: Graph
5. Cross platform testing coverage and results: Graph
6. 3d-party integration testing results: Graph or written explanation.
7. Testing and testing activities NOT done â describe reasons, connected risks, and recommendations.
8. Open Bugs:
All Open Critical and Blocker bugs, e.g.: 4 Critical&Blocker Bugs (list or link to jira)
Open Major bugs opened during the sprint, e.g: 40 Major bugs open during sprint. (link to jira)
9. Point to the buggiest product component and platform: Graph and explanation
10. General Conclusions
www.qaexperts.pro
55. Regression testing reports, daily or by cycles
Reports about Regression testing, cross platform testing, 3d-party integration testing, 3d party releases testing, etc.
1. Name of the Regression testing stage, e.g.: Regression Cycle 1 (Preliminary feedback)
2. Coverage and results: Graph
3. Open Bugs
New (created today) and All Open Critical and Blocker bugs:
New 4 Critical&Blocker Bugs, (All 5 Critical&Blocker Bugs) (list or links to jira)
New (Created today) and All Open Major bugs opened during the release:
New 10 Major bugs, (All Major bugs created for current release) (links to jira)
4. Specify Time needed to finish regression testing and/or necessity in additional regression and other testing.
5. Point to the buggiest product component, platform, or functionality if needed.
6. Recommendations
www.qaexperts.pro
56. Release Readiness testing Report (by necessity)
1. Specify functional testing accomplished and its status:
Regression and cross-platform testing,
3dparty integration testing,
UAT
e.g.: Written explanations and Links to Test management system,
or Graphs of testing coverage and results.
2. Testing and testing activities NOT done â describe reasons, connected risks and recommendations.
3. Specify Non-functional testing status, like performance and security testing.
Whether the product is ready or not ready from non-functional point of view.
Describe connected risks and recommendations.
4. Open Bugs.
All Open Critical and Blocker bugs, e.g.: 4 Critical&Blocker Bugs (list or link)
Open Major bugs opened during the release and All Major bugs, e.g: 40 Major bugs open during release.,
70 Major bugs are Open in general
(list or links to jira)
5. Functionality Not ready for release, etc.: (list or link)+ written explanation
6. General Conclusions!
www.qaexperts.pro
59. Reminder Reports (Be necessity)
1. Critical and Blocker Bugs in open state.
2. A Lot of Open Major bugs
3. A lot of Bugs and stories Not Done in the middle or later
stage of sprints: In development, in Test, etc.
4. Availability of not tested stories during stabilization period.
5. Inability to finish testing in time because of certain issues.
Reminder emails or chat messages about:
www.qaexperts.pro
62. Milestones for trends
⢠AffectedVersion â AffectedVersion field in jira
⢠Time Span - e.g. period between start and end
date of sprint
Time based conditions in jira queries:
a) createdDate >= '2017-08-30' and
createdDate <='2017-09-12 â
b) status changed TO 'READY TO TEST' after
'2017-08-30' before '2017-09-12â
c) assignee was in (âuserâ)
⢠Sprint â Sprint Field in jira, etc.
www.qaexperts.pro
65. Found on Environment
All bugs
Shows leaks of
bugs to later
development
stages
Pivot Report
www.qaexperts.pro
66. Bugs By Priority â Any Status
All Bugs
by
Aff. Version
Shows absolute
release quality
history
â decrease in
number of all
and critical bugs
shows process
improvements
Pivot Report
Note: Should be considered carefully when the team size and product increment increase.
Note: Blank version is bad, and it shows that the Affected Version is not specified for
certain bugs and thus they can be lost www.qaexperts.pro
67. Bugs By Priority â OPEN (not-closed)
All Bugs
by
Aff. Version
Shows still open
bugs for
releases
â Ideally No
critical and
blocker bugs
should be here
Pivot Report Note: If critical bugs are shown here for previous releases, they might be lost, forgotten or
blocked. www.qaexperts.pro
68. Bugs by RootCause
Typical Root causes of Bugs:
FrontendOthers
Backend
Frontend
+ Backend
Environme
nt
issues
3d party
components
www.qaexperts.pro
69. Bugs by RootCause
Critical
and Blocker
bugs
Shows Root cause of all bugs
Pivot Report
All Bugs
Shows Root cause of the most
of important bugs
www.qaexperts.pro
70. Created Vs Resolved - Cumulative
Critical
and Blocker
Bugs
Created should be less than resolved
Jira âCreated vs. Resolved Chartâ Gadget
Bug found vs. fixed chart is a defect analysis line chart that lets us see defect
removal process patterns and understand defect management effectiveness
www.qaexperts.pro
71. Created Vs Resolved - Cumulative
All Bugs
Difference should be minimal
and If it is big and no positive
trend is shown then probably
Bagathon sprint should be
planned
Jira âCreated vs. Resolved Chartâ Gadget
www.qaexperts.pro
72. Defects Dept
Critical and Blocker Bugs
Note: Should Be always close to Zero for Critical bugs
All Bugs
Spreadsheets + advanced Jira time period based queries
Defects dept is a cumulative difference between Resolved and Open bugs
= (Open in sprint N - Resolved in sprint N)+ (Open in sprint N+1- Resolved in sprint N+1 )+âŚ
Note: Shows accumulation of bugs in time
www.qaexperts.pro
73. Bugs Opened vs Stories Delivered
There should be correlation between the metrics - more stories you deliver more bugs may appear.
Ideal trend is to deliver more user stories with less bugs.
!!!Pay attention to anomalies and investigate them deeper.
Spreadsheets + Advanced Jira time period based queries www.qaexperts.pro
74. Bugs per Story Point
Critical
and Blocker Bugs
If number
decreases in
time it means
that we
improve QA
process Note: This metric is very good for cases when team size or
product increment is variable or change (increase)
All Bugs
Spreadsheets + Advanced Jira time period based queries www.qaexperts.pro
75. Bugs per Story Point
Other Variations:
1. Bugs per requirement
2. Bugs per developer
3. Bugs per 1000/Code
4. Time lost for bugs h - Time spent for Bugs vs time spent for stories
Note: Very useful to for making future project estimations.
www.qaexperts.pro
76. Fix Rate
Should be close to 100% or higher
Spreadsheets + Advanced Jira period based queries
Critical and Blocker Bugs
All Bugs
Fix Rate = Closed bugs / Open bugs
Ideally should be close to 100%
www.qaexperts.pro
77. âAverage Ageâ and âTime in statusâ
For Critical
and Blocker
Bugs
Jira âAverage Ageâ and âTime in Statusâ Gadgets
Average Age - Displays the average number of days issues have been unresolved.
For critical and Blocker bugs this time should be very short.
If process improves it should decrease in time.
Average Time in status, e.g. Open status
- Displays the average number of days issues have spent in status.
For critical and Blocker bugs this time should be very short.
If process improves it should decrease in time.
www.qaexperts.pro
78. Cumulative Flow Chart
Jira Agile Reports
To find process bottlenecks and predict project close time
www.qaexperts.pro
79. Bugs by Components/Platforms
Pivot or other spreadsheet Report
Bugs trend By Component â Critical and Blocker / All Bugs
Shows weakest component of the product, shows
where additional testing should be performed, shows
where technical Improvements should be made,
Shows whether component quality improves
Bugs trend By Platform â Critical and Blocker / All Bugs.
Shows amount of cross-platform bugs, shows the
weakest platform requiring more testing and
stabilization.
.
www.qaexperts.pro
80. Sprint and Business Metrics
Estimates Accuracy % = Estimated amount of
effort/Actual amount of effort-1
Story Delivery Rate % = Actual Stories Completed /
Committed Stories
% of Change Requests = Number of change requests /
Number of Initial Backlog stories
Change Requests Costs = Time spent of change
requests * Average Rate of developers on the project
% Costs of Defect = Average Time for defect fix *
Average developer rate on the project
ď§ Trends
ď§ For Sprint
ď§ General value
www.qaexperts.pro
82. User Satisfaction
Number of hot fixes had to be made in this or that release.
Shows leaks of Critical bugs on prod.
Bugs found by end-users
Critical/Blocker (Shows leaks of Critical bugs on prod.)
Major and Low priority
Customer Complaints and Feedbacks reports. (Net Promoter Score)
A/B Testing results
Shows which functionality is more preferred
Other Marketing investigations and surveys
All these answer the following questions:
a) What are the weakest points of the product?
b) What functionality is most used ?
c) Which focus in product development and testing should be made?
www.qaexperts.pro
83. Qualitative Metrics about the project state
If you want to gather more metrics about project processes, approaches,
technology, communication, mood in teams, etc.
Just ask people to provide evaluation and feedback
Experts evaluation and/or Audit
Surveys
Retrospectives
1-2-1 and other meeting
Rumors/small talks at Smoking room/kitchen ď
www.qaexperts.pro
85. Developers and Dev Team efficiency
REOPENED:
Reopened Bugs Ratio % = Reopened bugs vs Resolved bugs
Reopened Bugs Ratio % = Reopened Stories vs Stories
delivered to testing
STORY POINTS DELIVERED
STORY DEFECTS ISSUED
STORY DEFECTs/STORY POINTs rate %
QUALITATIVE METRICS: Gather qualitative metrics about
developers: approach to coding, complexity of code,
contribution to innovation, communication efficiency, quick
reaction to problems, etc. ď§ Per PROJECT in general
ď§ Per Team
ď§ Per specific Developer
www.qaexperts.pro
86. QAs and QA team Efficiency
REJECTED
Absolute Number of rejected bugs:
All resolutions,
Not a bug,
Duplicate
Cannot reproduce
Incomplete
Rejected Rate % = rejected vs created
BUGs LEAKS: to PROD other later stages of development.
Break down by
Per product part
Per functionality
So you can correlate it with a certain QA or group of QAs
Separately:
ď§ For critical and blocker
ď§ For Lower priority
defects
ď§ Per PROJECT in general
ď§ Per QA Team
ď§ Per specific QA
www.qaexperts.pro
87. QAs and QA team Efficiency
WORK DONE:
ďą Stories Tested
ďą Created bugs Critical and Blocker (not rejected)
ďą Created bugs Major and lower priority (not Rejected)
ďą Test cases Created (passed DoD). Test cases rejected.
ďą Test cases Executed.
ďą Automation scripts Created (passed DoD). Test cases
rejected.
ďą Advanced test artifacts created and activities
accomplished: test suites, plans, release support, etc.
QUALITATIVE METRICS: Gather qualitative metrics about QAs:
Complexity of tasks, Communication efficiency, Proactivity,
Contribution to innovation, Test management skils, Conflict solving,
etc.
ď§ Per PROJECT in general
ď§ Per QA team
ď§ Per specific QA
ď§ For Sprint (Velocity)
ď§ Trends
ď§ General value
www.qaexperts.pro
90. ďą Unit test coverage & number of test
ďą API testing coverage & number of test
for product in general
by component
ďą GUI Automation testing coverage & number of test
ď§ for product in general
ď§ by product part
ď§ by platform/device
ď§ by type of test
end-to-end test
Smoke test
Regression test
UAT test
ď§ by test priority:
Priority 1
Priority 2
Priority 3
Automation Coverage & Results
www.qaexperts.pro
91. Automation Efficiency
AUTOMATION SPEED:
ď§ Automation tests Frequency
ď§ Duration of tests Running
ď§ Velocity of script creation and maintenance
ď§ Time for script creation and maintenance
AUTOMATION EFFICIENCY:
Stability of automation
False Positives
Root Cause for failure
Bugs found by Automation
Critical and Blocker
Other bugs
Bugs missed with automation.
Critical and Blocker
Other bugs
ROI â Time automation economy, etc.
Cost per automated test
ď§ Per PROJECT in general
ď§ Per QA team
ď§ Per specific QA
ď§ For Sprint (Velocity)
ď§ Trends
ď§ General value
www.qaexperts.pro
94. Tools
ď§ Jira + Special time-
related JQL
ď§ Confluence + Jira and
other Macros in it
ď§ Spreadsheet processors:
Googlesheets, Excel
ď§ TestTrail
www.qaexperts.pro
95. JiraReportingInGsheets Tool
Link to Help:
http://jirareportingingsheets.qaexperts.pro/
JiraReportingInGSheets free add-on
for GoogleSheets
allows to automatically build custom jira reports
in google sheets
Created by (QAExperts.pro):
www.qaexperts.pro
Volodymyr PrymakovPilvinskyi Dmitry