Website SIGIST: TMMi, the successor of TMM, was first published in 2008. Since then, a lot of organisations have used it to assess or improve their test processes. Jan Jaap Cannegieter used TMMi in Telecom, public services, finance and utility. Based on over 25 assessments conclusions about the level of testing in The Netherlands can be drawn: - Which areas of testing are well developed? - Which areas of testing are not well developed? - What conclusions can be drawn based on these data? These questions will be answered in the presentation. The main conclusion will be that those areas that can easily be influenced by testers are quite well developed; test process improvement is something done by testers in their cave! The areas where other stakeholders need to participate are not that well developed. So to improve testing we need to involve other stakeholders better. The way we should act to involve the relevant stakeholders will be explained and analyzed. Jan Jaap Cannegieter is a leading test, QA, CMMI and requirements expert in the Netherlands. He has 20 years of experience in ICT, starting with software testing and quality assurance in ICT-projects. In the last few years he has completed several TMMi test process improvement, CMMI, SPI and requirements projects. In addition to his assignments, Jan Jaap is Vice President of SYSQA B.V., a company of 180 employees specializing in requirements, software testing, quality assurance and software process improvement. Jan Jaap is the writer of several articles and books, including The little TMMi. Jan Jaap was a member of the TMMi development team and is TMMi accredited lead assessor.

1. Tester, get out of your cave!
Jan Jaap Cannegieter –
Vice President SYSQA B.V.
 jcannegieter@sysqa.nl
@jjcannegieter

2. 2
© SYSQA Almere

3. Why are we here?
© SYSQA Almere
3

4. Agenda
• TMMi in under 10 minutes
• Data from 20 TMMi workshops
• Conclusions
• How to get out of your cave
© SYSQA Almere
4

6. TMMi – essentials
• Test maturity framework
• Benchmark model for assessments
• 5 levels
• Based on:
• TMM
• Evolutionary Testmodel (Gelperin / Hetzel)
• CMMI for Development
• In line with ISTQB terminology
• Developed by the TMMi-foundation
• Public domain
© SYSQA Almere
• www.tmmifoundation.org
6

7. TMMi levels
Defect Prevention
5. Optimization Test Process Optimization
Quality Control
Test Measurement
4. Management
Software Quality Evaluation
and Measurement Advanced Peer Reviews
Test Organization
Test Training Program
3. Defined Test Life Cycle and Integration
Non-Functional Testing
Peer Reviews
Test Policy and Strategy
Test Planning
2. Managed Test Monitoring and Control
Test Design and Execution
© SYSQA Almere
Test Environment
1. Initial
7

8. How I got my data
• Goals
• Learn
• Presales
• Assignments
• 20 organizations
• Telecommunication: 2
• Government: 7
• Financial services: 9
• Industry: 2
© SYSQA Almere
• Analysis
8

9. Results
Fasten your seatbelts

10. Test policy and strategy - goals
• Establish a test policy
• Establish a test strategy
• Establish test performance indicators
© SYSQA Almere
11

11. Test policy and strategy: A: 30% V: 14%
Test policy and strategy
100
Percentage of specific goals met
90
80
70
60
50
40
30
20
© SYSQA Almere
10
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Organizations
12

12. Test planning - goals
• Perform a product risk assessment
• Establish a test approach
• Establish test estimates
• Develop a test plan
• Obtain commitment to the test plan
© SYSQA Almere
13

13. Test planning: A: 41%, V: 13%
Test planning
Percentage of specific goals met
100
90
80
70
60
50
40
30
20
© SYSQA Almere
10
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Organizations
14

14. Test monitoring and control - goals
• Monitor test progress against plan
• Monitor product quality against plan and
expectations
• Manage corrective action to closure
© SYSQA Almere
15

15. Test monitoring and control: A: 47%, V: 15%
Test monitoring and control
Percentage of specific goals met
100
90
80
70
60
50
40
30
20
© SYSQA Almere
10
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Organizations
16

16. Test design and execution - goals
• Perform test analysis and design using test design
techniques
• Perform test implementation
• Perform test execution
• Manage test incidents to closure
© SYSQA Almere
17

17. Test design and execution: A: 60%, V: 8%
Test design and execution
100
Percentage of specific goals met
90
80
70
60
50
40
30
20
© SYSQA Almere
10
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Organizations
18

18. Test environment - goals
• Develop test environment requirements
• Perform test environment implementation
• Manage and control test environments
© SYSQA Almere
19

19. Test environment: A: 59%, V: 9%
Test environment
Percentage of specific goals met
100
90
80
70
60
50
40
30
20
© SYSQA Almere
10
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Organizations
20

20. Overview
Per process area
100
90
80
70
60
50
40
30
20
10
0
© SYSQA Almere
21

21. My conclusion
Outside 30%: Test policy and strategy
41%: Test planning
47%: Test monitoring and control
59%: Test environment
© SYSQA Almere
Inside 60%: Test design and execution
22

22. We are in a cave!
© SYSQA Almere
23

23. © SYSQA Almere
24

24. In case you disagree…
© SYSQA Almere
25

25. Best practices ‘outside of your cave’
• Perform a product risk assessment and establish a test
approach with relevant stakeholders
• Obtain commitment for your plan from relevant stakeholders
• Plan stakeholder involvement in your project
• Monitor stakeholder involvement in your project
• Make release advice with people outside of testing
• Establish and maintain a test strategy with relevant
management
• Define, measure and report test performance indicators to
relevant stakeholders
© SYSQA Almere
• Establish and maintain a test policy with higher management
(business and IT)
26

26. Stakeholder management
The basics
27

27. Stakeholder
© SYSQA Almere
A person or group of persons who will be involved in or
effected by your test project or improvement project
28

28. Stakeholder management
• Stakeholder identification
• Stakeholder analysis
• Stakeholder involvement
© SYSQA Almere
29

29. Stakeholder aspects
• Organizational level
• Power
• Business interest
• Mindset
• Ambition
• Behavioral style
• Stakeholder type
© SYSQA Almere
30

30. Type of stakeholders
Power
Sleeping
Dangerous Dominant
Definitive
Urgency Demanding Discretionary Legitimacy
© SYSQA Almere
Dependent
Mitchell, Agle, Wood, 1997
31

31. Stakeholdercard – an example
© SYSQA Almere
32

32. Please think this over
I’ll turn on some inspiring music

33. © SYSQA Almere
34

34. © SYSQA Almere
35

35. Stuart Reid’s test competence model for
testers
Test skills
Soft skills IT skills
© SYSQA Almere
Domain
knowledge
36

36. My test competence model for test
consultants & test managers
Test skills
Political
IT skills
skills
© SYSQA Almere
Domain
Soft skills
knowledge
37

37. © SYSQA Almere
38

38. More interesting graphs

39. TMMi level by percentage
84%
L1
L2
L3
L4
L5
© SYSQA Almere
10%
6%
TMMi Level
40

40. CMMI vs. Non-CMMI
80
70
60
50
40 CMMI
30 no CMMI
20
10
© SYSQA Almere
0
TPG TP TMC TT TO
41

41. TMMi score by Industry
70
60
50
40 Government
30 Finance
Industry
20
10
© SYSQA Almere
0
TPS TP TMC TDE TE
Note: Telecom weg gelaten slechts 2 data punten 42

42. Formeel vs. Informeel
14%
Formeel
Informeel
© SYSQA Almere
86%
43

43. Email for the stakeholder card:
jcannegieter@sysqa.nl
© SYSQA Almere
45

44. © SYSQA Almere
46