Доклад Дмитрия Гуменюка на конференции SQA Days-20. 24-26 ноября 2016. Минск www.sqadays.com

1. Software quality assurance days
20 Международная конференция
по вопросам качества ПО
sqadays.com
Минск. 24–26 ноября 2016
Дмитрий Гуменюк
EPAM Systems. Минск, Беларусь
ReportPortal.io: пример CI/CD c метриками на
основе продукта с Открытым Кодом

2. http://ReportPortal.io
пример CI/CD с метриками, на основе продукта с Открытым Кодом
Проект Менеджер
10 лет с компанией EPAM Systems.
Разработчик в прошлом
Возглавляю разработку инициатив внутри
Центра Компетенций Тестирования.
Report Portal – Product Owner.
DZMITRY HUMIANIUK
ДМИТРИЙ ГУМЕНЮК

3. http://ReportPortal.io
пример CI/CD с метриками, на основе продукта с Открытым Кодом
Проект Менеджер
10 лет с компанией EPAM Systems.
Разработчик в прошлом
Возглавляю разработку инициатив внутри
Центра Компетенций Тестирования.
Report Portal – Product Owner.
DZMITRY HUMIANIUK
ДМИТРИЙ ГУМЕНЮК
Проект Менеджер
10 лет с компанией EPAM Systems.
Разработчик в прошлом
Возглавляю разработку инициатив внутри
Центра Компетенций Тестирования.
Report Portal – Product Owner.

4. http://ReportPortal.io
пример CI/CD с метриками, на основе продукта с Открытым Кодом

5. http://ReportPortal.io
пример CI/CD с метриками, на основе продукта с Открытым Кодом
OPEN
SOURCED
GPL v3.0
GITHUB.COM/ REPORTPORTAL

6. http://ReportPortal.io
пример CI/CD с метриками, на основе продукта с Открытым Кодом

7. http://ReportPortal.io
пример CI/CD с метриками, на основе продукта с Открытым Кодом
МЕТРИКИ

8. МЕТРИКИ “ПОЖАРНЫЕ ИЗВЕЩАТЕЛИ”
Name Why it is important to monitor
this
What decision can be made Periodi
city
Calculation
Execution
Frequenc
y
To understand and evaluate
how often test cases executed.
Goal: Increase execution ratio
per day/build.
Configure automated execution
of TA after commits (smoke,
acceptance) and regressions
right after nightly builds.
Triggers
1 execution per week or less
(???)
Target
1 per commit
Interpretation
code change should be
verified not to affect
functionality or other code
base.
Each
day
Measures
1. test executions count
Formula
Unit of measure
Number
% of
results
analyzed
To understand and evaluate
percentage of reviewed fail
reports. If fail report not
reviewed - there is not any
value
Triggers
<60%
Target
100%
Interpretation
failed report(test cases)
should be review
Each
day
Measures
1. No of not analyzed failed test cases (TI)
2. No of failed due to product bug (PB)
3. No of failed due to automation issues
(AB)
4. No of failed due to system issue (SI)
Formula
100% - To_Investigate / (PB+AB+SI+TI)
Unit of measure
%
http://ReportPortal.io
пример CI/CD с метриками, на основе продукта с Открытым Кодом

9. МЕТРИКИ “ПОЖАРНЫЕ ИЗВЕЩАТЕЛИ”
Name Why it is important to
monitor this
What decision can be made Period
icity
Calculation
% of
product
issues
(stability of
automatio
n)
To understand and
evaluate percentage of
failed test which
corresponds to
production issues
Triggers
trend not being positive
Target
>90%
Interpretation
trend of fails due to test automation
instability should decrease (automation
issues, system issue, not analyzed failed
test cases)
Measures
1. No of not analyzed failed test cases (TI)
2. No of failed due to product bug (PB)
3. No of failed due to automation issues (AB)
4. No of failed due to system issue (SI)
Formula
PB / (PB+AB+SI+TI)
Unit of measure
%
% of
automatio
n issues
To understand and
evaluate percentage of
failed test which
corresponds to test
automation issues (code
issues, test case invalid)
Triggers
trend not being positive
Target
<5%
Interpretation
trend of fails due to test automation
instability should decrease
Measures
1. No of not analyzed failed test cases (TI)
2. No of failed due to product bug (PB)
3. No of failed due to automation issues (AB)
4. No of failed due to system issue (SI)
Formula
AB / (PB+AB+SI+TI)
Unit of measure
%
http://ReportPortal.io
пример CI/CD с метриками, на основе продукта с Открытым Кодом

10. МЕТРИКИ “ПОЖАРНЫЕ ИЗВЕЩАТЕЛИ”
Name Why it is important to
monitor this
What decision can be made Perio
dicity
Calculation
% of
system
issues
To understand and
evaluate percentage of
failed test which
corresponds to system
issues (no connectivity,
lack of resources and
etc.)
Triggers
trend not being positive
Target
<5%
Interpretation
trend of fails due to system issue
instability should decrease
Measures
1. No of not analyzed failed test cases (TI)
2. No of failed due to product bug (PB)
3. No of failed due to automation issues (AB)
4. No of failed due to system issue (SI)
Formula
SI / (PB+AB+SI+TI)
Unit of measure
%
Execution
Time
To understand and
evaluate time of each
execution cycle.
Triggers
Regression cycles > 15h
acceptance cycles >1h
Target
regression <8h
acceptance <1h
Interpretation
time for execution should not go
over described limits
Measures
1. Execution duration of regression, hours (Reg)
2. Execution duration of acceptance, hours (Acp)
Formula
Reg < 15h
Acp < 1h
Unit of measure
Hours
http://ReportPortal.io
пример CI/CD с метриками, на основе продукта с Открытым Кодом

11. http://ReportPortal.io
пример CI/CD с метриками, на основе продукта с Открытым Кодом
ReportPortal.io Community
M eet-up
14 December 2016
19:00
Imaguru, M insk

12. http://ReportPortal.io
пример CI/CD с метриками, на основе продукта с Открытым Кодом
http:/ / ReportPortal.io
GITHUB.COM / REPORTPORTAL
Report Portal Community
ReportPortal.io Community
ReportPortal_io