This document summarizes the journey of defining and automating QA reports and metrics for a growing company handling multiple projects simultaneously. It describes how initially manual reporting was automated using an external tool to gather data from sources like JIRA and SonarQube to generate reports on Confluence. Challenges included different client tooling, prioritizing reports after releases, and tool maintenance. Over time, the experienced team focused less on governance and more on consulting. Metrics are now used selectively to measure process improvements rather than compliance. QA status is included in high-level project reports, with the team deciding if detailed QA reports are needed based on the project.
3. Agenda
3
01 The Beginning
02 An unexpected journey / Define QA Reports and metrics
03 Find your Black Arrow / Automate QA Reports
04 The battle of five army / Challenges
05 And back again / Few years later
5. Bigger, bigger and bigger
• New clients from pharmacy, insurance and banking
• Which required more documentation and more formal processes
• Do not confident with agile methodologies and exploratory testing approach
• Project duration from 2 to 6 months
• Fast growing company
• A lot of newcomers which do not understand our ways of working
• Inexpierence team
5
6. How to control several simultanuously projects?
• Governance on company level
• Deliver always high / expected quality
• Support inexperienced team / share knowledge
• Reporting projects status to Company Board
• Compare projects
• Identify good practices
• Understand project complexity
6
8. How to identify metrics?
• Metrics should encourage to ask questions not give the answers
• Measure the processes not people
• Qualitative and Quantitave
• Highlight important processes, activities, ways of working
• Each release has quality report
8
9. 9
1 x full width image & text
placeholder (no header bar)
10. 10
1 x full width image & text
placeholder (no header bar)
11. 11
1 x full width image & text
placeholder (no header bar)
12. 12
1 x full width image & text
placeholder (no header bar)
13. 13
1 x full width image & text
placeholder (no header bar)
14. 14
1 x full width image & text
placeholder (no header bar)
15. 15
1 x full width image & text
placeholder (no header bar)
26. Adulthood
• Initial objectives are not valid anymore
• Experienced team
• Which know what is important for us
• Which gained the clients and board trust
• From governance to consulting
• QA status as part of project reports
26
27. Be reliable, do not measure, just do it
• Test Automation coverage dilemma
• DoD ensure me that I automated necessary tests on different levels
• Why I should measure the metric if I knew the result?
27
28. Now
• Project’s team makes decision if QA Report is needed and choose
metrics
• Most popular case: only final QA Report before UAT (are we production
ready?)
• Measure process improvement
• For the last proces change (shift left) we measured the improvement rate
• Eg. as a result we measured that Internal Quality raised from 50% even to
90% acceptance rate
28
29. The Ring / QA Reports
• Transform successfully
• Learn how to deliver bigger projects for demanding clients
• Identify what is important for us
• Teach newcomers our ways of working
• And we have a tool for next transformation
29