Software Testing meetup within "Open Technology Week" by Sigma Software, Odesa 2017.

1. Using the test process improvement
models. Case study based on TPI
NEXT model
Anton Muzhailo,
Senior Python Automation Engineer, GlobalLogic
Certified ISTQB Test Manager

2. About me
Anton Muzhailo
Senior Python Automation Engineer, GlobalLogic
• ISTQB Certified Test Manager
• ISTQB Certified Trainings Coach in CodeSpace.com.ua
• 3 years of mentoring experience, 250+ students
in/muzhailo/
Odessa, Sep 6, 2017

3. • Can you improve your testing process on your project? Should you?
• Existing models. How to choose the best one in your case
• A good place to start and integrate the Test Process Improvement activities
• Going through the assessment and implementation stages of the TPI Next
test process improvement model integration
Agenda

4. “Continuous improvement of the quality, and the efficiency of the
testing process, in the context of the total software process”
- Costs
- Effort
- Time
- Early defects
- Risk management
- Coverage
- Product quality
- Predictability
- Control
- Development Model
- Project Planning
- Requirements
- People
What is test process improvement?
One more theoretical useless thing, huh?

6. What is the quality?
Is the quality of
this product high?
What is the
quality of that
product?
Which product is
better in terms of
quality?

7. Product quality
Quality is a degree to which a component, system or process meets specified
requirements and/or user/customer needs and expectations.
What is the quality of that product?
Is the quality of this product high?
Which product is better in terms of quality?
Q(%) =
# requirements satisfied
total
x 100
Acceptance criteria list  good_enough_point
if Q ≥ good_enough_point:
return True
Qproduct1 Qproduct2

8. Product
quality
Development
technology
Process
quality
Money,
scope,
schedule
People
quality

9. How development technology can be improved?
1. Ask yourself, why do you think it can be improved. (HINT: It can always be improved)
2. Raise the question to developers. Explain how they are responsible for quality.
3. Involve project manager if needed
4. Wait for questions why does current technology is bad
5. Expect severe resistance even if you have strong arguments
6. Fight to the death or leave with nothing
Сollect the evidences of a poor
technology.
Introduce the metrics that will
reveal the problemsOR

10. How people quality can be improved?
Trainings
Fire & hire
another
Do the job
for them

11. How to improve money, schedule and scope?

12. How the process can be improved?
Process
characterictic
Descripton
Visibility Does the left hand know what the right hand is doing?
Understandability Is it easy for a newcomer to get into? To what extent is the process
explicitly defined?
Acceptability Is the process acceptable and usable by engineers?
Rapidity What ASAP means in terms of your project? How fast you could
reach the goal?
Robustness Can the process handle the unexpected problems and adapt?
Reliability Is the process able to catch the process errors before they result in
product errors?
Adequacy Is every activity in the process has a good reason to happen?

13. With all that said, can you improve your
product quality?
A. Yes B. No
C. Dunno D. What?

14. Test process
Improvement types
Model-based
approaches
Process referenced
Model-referenced
Analytical
approaches
Casual analysis
GQM (goal,
question, metric)
Retrospectives

15. Existing Test Process Improvement Models
TMM
TPI
CTP
STEP
TOM
TIM
SQR
TMap
TAP TSM
TMM – Test Maturity Model
STEP – Systematic Test Evaluation Process
CTP – Critical Test Processes
TPI – Test Process Improvement
TMap – Test Management approach
TSM – Testability Support Model
TAP - Testing Assessment Programme
TIM – Test Improvement Model
TOM – Test Organization Maturity
SQR – Software Quality Rank

16. TPI Next Areas and Levels
Stakeholder
Relations
1 Stakeholder commitment
2 Degree of involvment
3 Test Strategy
4 Test Organization
5 Communication
6 Reporting
Test
Management
7 Test Process Management
8 Estimating and Planning
9 Metrics
10 Defect Management
11 Testware Management
Test
Profession
12 Methodology Practice
13 Tester Professionalism
14 Test Case Design
15 Test Tools
16 Test Environment
# Level Meaning
1 Initial
Chaotic, ad-hoc
processes
2 Controlled
Doing the right
activities
3 Efficient
Doing the
activities right
(effective)
4 Optimizing
Continuously
improving the
activities

17. TPI Next
I
(Initial)
II
(Controlled)
III
(Efficient)
IV
(Optimizing)
16 key areas
key areas
maturity levels
checkpoints
16
4
157

18. TPI Next maturity matrix example
http://reqtest.com

19. TPI Next implementation process
Evaluate current
situation
Plan this
improvement
cycle
Implement
according to the
plan
Evaluate the
results
Plan another
improvement
cycle

20. Case Study on implementing TPI Next
1) Get the tools at http://www.tmap.net/tpi-downloads
2) Evaluate current situation and filling the template
3) Observe the “big picture” – test maturity matrix
4) Plan the needed improvements by creating Test Improvement Plan
5) Implement them according to the plan
6) Evaluate again and figure out what’s went good and what’s not
7) Plan the next improvement cycle

21. Excel workbook that:
• has 16 sheets for
each key area
• Each area has
maturity levels
• Each maturity level
has its checkpoints
• Each checkpoint has
Y/N flag to select
Manual with all the magic
explained deeper
1

22. 2
True/False
flag to select

23. 3
Better to leave
comments why
checkpoint is
not reached
Expect such
references. They will
show the root
problems on project

24. 4
Test area
groups
Use to prioritize
(High, Neutral,
Low)
Checkpoint
clusters, not
covered here

25. 5
Those unfilled
checkpoints are our
first goal to satisfy!
We can’t deal with this one
as we are not passed the
Controlled level yet!
For now, assume that all
unfilled checkpoints are
equal within one level!

26. 6
Priorities has been set
Now it’s time to create the test process
improvement plan based on priorities
The principal
stakeholder agrees
with the documented
test strategy
Test activities are started early, timely
before test execution, with the goal to
keep the test activities of the project’s
critical path
A tester is involved in the
analysis and mitigation of
project risks
Each test case is related
to a test basis document
in a transparent way
Test environment
requirements are
documented

27. Priority Checkpoint Implementation way Owner
1
The principal stakeholder
agrees with the
documented test strategy
Conduct formal walkthrough review of a test
strategy. Explain the test strategy is project-
dependent. May require the involve of upper
management.
Nilson
1
Each test case is related to
a test basis document in a
transparent way
Evaluate if test management tool is able to create
the traceability matrix. Make sure there is a
formal management of test basis.
Nilson
2
A tester is involved in the
analysis and mitigation of
project risks
Ask if there are any objections to involve tester to
project risks analysis. Get an aggreement.
McGower
2
Test activities are started
early, timely before test
execution, with the goal to
keep the test activities of
the project’s critical path
Communicate with Nilson. Clear and aggreed test
strategy should solve this problem. Check the test
plan too.
Frattia
3
Test environment
requirements are
documented
Document all the test environments and get the
approval. Establish the formal process of test
environments documentation
Alvarez
7

28. Implement the improvements

29. 9
Was not implemented due to severe resistance
from management as they think that project risks
should only be handled by management. May
require audit or independent assessment to solve
4 of 5 improvements were
implemented successfully
A tester is involved in the
analysis and mitigation of
project risks
Time to plan the next improvement cycle!

30. Thanks!
No questions, right? 
in/muzhailo/
Odessa, Sep 7 2017
Anton Muzhailo