how to improve R&D process- presented during TACT testing metrics convention

1. A successful improvement process with
measurable results
Yigal Cohen
(Ex) R&D Manager Deputy - NDS
March 2010 – ‫אדר תש"ע‬

2. Chosen by the top TV operators
– Global company
– Complex
– industry leading system
– Changing matrix

3. Improvement plan goals
• What ?
– Increase business results by improving project
management, process & methodologies
• How?
– Provide a set of principles, tools, metrics, and
improvement framework
– Push Lines to lead and manage their actual
improvement plan
– Accompany and mentor real projects
3

4. Continuous Improvement Framework
4
3
2
1
0
Criteria
R&D Charter
Toolkit
14 Ch.14 Every Build will have the following 5: NA
artifacts:1. Build in Release management
system2. Should have both production as
well as Debug enabled image3. Release note
with CQ list and component details
15 Ch.12 a Defects and issues are tracked 2 : somewhat
16 Ch.12 b Defects and issues undergo root-cause 2 : somewhat
analysis
17 Communication plan is defined 3 : well into it
18 Ch.13 Artefacts are held within a configuration or 5: NA
document management system
19 Ch.15 Project has end of milestone reviews 2 : somewhat
20 Communication about the completion of 2 : somewhat
milestone
Self Assessments
Self Assessment Analysis Improvement Plan
4

5. The Community
8 WW_QA + 440
35 SQAs 2200
R&D Management AI Holders
Users
5

6. Local toolkits
R&D 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Division 2 4 7 10 11 12 14
Line 2 6 11 13 14
Project
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Every division and line is authorized and encouraged to prepare its own
procedures for part or all of the R&D Charter.
The most specific procedures will be used for each project.
6

7. SDP Coding
• Time & effort Estimation • Re-factpriong
• Scheduling • Complexity analysis
• WBS • Static Analysis
• Design/Dev/Test done
iteratively
• Progress Monitoring
Configuration
• Burn Down • Branching Policy
• TOC • Building
• Risk Management • Automated Unit level tests
Requirements Testing
• Tests cover requirements • Common Logging
• Automated regression tests
Design • TPI
• Defensive Design & coding
• UML
Integration
• Pattern Design • Continuous Integration
• Release
• Release Criteria
7

8. • Does it help ?
• To Whom ?
8

9. One line - improvement Done Successfully
activities
Infrastructure
• Coding standards written for New Generation Smart Card, SimCAM QC
Tools, and Non-Smart Card Products
• Migration to SVN source management
• Root cause analysis accepted part of standard quality procedure
• Smart Card development utilizing Visual Studio environment and
associated tools (pilot)
Actual improvements
• New Smartcard code base developed using:
– Unit test on 80% of code
– Continuous Build on 100% of code
– ->New Smartcard ROM integration & testing in only 4 weeks!
• SVN configuration management (Most of the line)
• Root Cause Analysis being performed on all defects found in I&T and
at the customer site
• Average R&D Charter compliance of 89%
9

10. Slide 9
AV4 Not sure where this fits in.
Dr Alison Vincent, 02/07/2008

11. Impacts on the development process
(Saves time, better quality and increase efficiency)
• Integration Plan
– Planning the integration saves a lot of time by making pre-
integrations and preparing tools in advance (Saves time)
• Test Strategy
– Testing process was improved especially in multi-site projects
(Saves time and improve quality)
• Architects started to increase their involvement along
the development process (Quality and efficiency)
• Spreading of Coding Standards, Build, Root Cause
Analysis and Lessons learned across R&D with
Common global language and transparency (All)
• Contributed to increase profit by 25% in a year!
10

12. Customer defects reported in CQ
16000
14000
12000
10000
8000
6000
4000
2000
0
6
6
6
6
7
7
7
7
8
8
8
8
9
'0
'0
'0
'0
'0
'0
'0
'0
'0
'0
'0
'0
'0
Customer Defects
1
2
3
4
1
2
3
4
1
2
3
4
1
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
Q
900
800
700
600
500
400
300
200
100
0
Qtr 1 Qtr 2 Qtr 3 Qtr 4 Qtr 1 Qtr 2 Qtr 3 Qtr 4 Qtr 1 Qtr 2 Qtr 3 Qtr 4 Qtr 1
'06 '06 '06 '06 '07 '07 '07 '07 '08 '08 '08 '08 '09
11

13. R&D Charter
NDS R&D Charter
is the guideline for the process
1. For Planning, Tracking & Management projects will have:
1. Software Development Plan
2. Resource Allocation plan
3. Re-use strategy
4. Test Strategy and Integration Plan
2. Development phases
1. Requirements will be Documented/Reviewed/Managed
2. Designs will be documented and Reviewed
3. Coding standard
4. Code review
5. Automatic tools and ATP for every API
3. Infrastructure
1. Artefacts will be held within a configuration or document
management system
2. Defects and Issues will be tracked and will undergo root cause
analysis
3. Projects will have end of project reviews
4. Builds will be automated and repeatable
5. Released builds will be controlled
12

14. What’s in the toolkit ?
Templates
Other Refs Checklists
Guidelines
Training
Examples
Support
Tools
Contacts
!
13

15. R&D Charter Toolkit
14

16. An Example Criteria for Self
Assessment
4: TS updated throughout
project
3: TS compliant to the checklist
communicated and followed
2: TS reviewed and approved
1: Test Strategy is written
0: No Test Strategy
15
15

17. Self Assessments - Example
Ch.01 Projects will have a Software / 2: Documented or presented at 2 http://shareatnds.in.nds.com/sites/Intranet/Op
Hardware Development Kick Off meeting enTV/master_ui/Shared%20Documents
Plan that meets a defined /project_plan.aspx
schedule according to the
customer's needs
Ch.02a Projects will have a Test 2: Approved Test Strategy 2 http://doogle.uk.nds.com/worksitemp/dispatch?obj
strategy (SQSC-style) reviewed ectId=%21V3%21UKDMPRODUCTION1%
and approved by all 21C%21PJ%2441860%21&returnUrl=1211
stakeholders 45625069359336&operation=zoom&dataso
urce=project
Ch.02b Projects will have an Integration 0: No Integration Plan 0 Include link to plan
plan
Ch.03 Projects will have clearly defined 4: All roles actively performed 4 http://shareatnds.in.nds.com/sites/Intranet/Op
roles and responsibilities enTV/master_ui/default.aspx
Ch.04 Projects will agree on the 1: Initial discussion on re-use 1
strategy for re-use at of previous assetts
Project Kick-Off
(requirements, design,
test and code)
Ch.05 Projects will follow the R&D 2: resource plan done 2 http://shareatnds.in.nds.com/sites/Intranet/Op
Resource allocation plan (available) but not enTV/master_ui/Shared%20Documents
reviewed against /project_plan.aspx?RootFolder=%2fsite
delivery requirements
s%2fIntranet%2fOpenTV%2fmaster%5f
ui%2fProject%20Plan%2fWorkPlan&Fo
lderCTID=&View=%7b69BD6617%2d6
CCB%2d46E0%2dA5CE%2dFCE21C8
85064%7d
Ch.06 Requirements will be 2: requirements and change 2 http://doogle.uk.nds.com/worksitemp/dispatch?dat
documented / managed / requests documented asource=documentFolder&operation=zoom
reviewed (Document, ReqPro, &objectId=!V3!UKDMPRODUCTION1!C!FD
ClearQuest). $44881!&returnUrl=12114569796644311
Ch.07 Designs will be documented & 3: design reviews done with 3 http://doogle.uk.nds.com/worksitemp/dispatch
reviewed tangible outcomes, key ?datasource=documentFolder&operatio
design choices n=zoom&objectId=!V3!UKDMPRODUC
documented and major
TION1!C!FD$44875!&returnUrl=12114
issues fixed.Traceability
and effectiveness
5717958875383 16
checked in review

18. Sample
Charter Charter Item P1
Analysis
P2 P3 P4 Average
Item #
Ch.01 Projects will have a Software / Hardware Development Plan that meets a defined schedule
according to the customer's needs 1 2 2 2 1.8
Ch.02a Projects will have a Test strategy 3 1 2 0 1.5
Ch.02b Projects will have an Integration plan 4 3 3 3 3.3
Ch.03 Projects will have clearly defined roles and responsibilities 1 2 4 4 2.8
Ch.04 Projects will agree on the strategy for re-use at Project Kick-Off (requirements, design, test
and code) 1 4 2 1 2.0
Ch.05 Projects will follow the R&D Resource allocation plan 4 3 4 1 3.0
Ch.06 Requirements will be documented / managed / reviewed 2 2 2 3 2.3
Ch.07 Designs will be documented & reviewed 3 1 4 2 2.5
Ch.08 Code will follow a coding standard 2 3 3 3 2.8
Ch.09 Code will be reviewed 2 2 1 1 1.5
Ch.10 Builds will be automated and repeatable 3 4 3 3 3.3
Ch.11 APIs will have a tool by which they can be tested 1 1 1 1 1.0
Ch.12a Defects and Issues will be tracked 3 3 3 3 3.0
Ch.12b Defects will undergo root cause analysis 2 2 2 1 1.8
Ch.13 Artefacts will be held within a configuration or a document management system 4 4 4 4 4.0
Ch.14 Released builds will be controlled 2 3 3 4 3.0
Ch.15 Projects will have end of project reviews 3 2 3 0 2.0
COLUMN AVERAGES 2.4 2.5 2.7 2.1 2.4 17

19. Improvement Plan – 3 Examples
• Faster and more reliable builder
– Learn Hudson and build infrastructure. 6 weeks. Danny
– Run a pilot. Next 8 weeks. Mike.
– Evaluate Results. Next 2 weeks. Rachel
• Deploy Scrum in all projects
– Use Scrum in projects X, Y, Z. Q1. Danny, Shani, Rafi
1200 action items from IL/FR/IN/UK/USWeek 3
– 1 Day seminar for GL, including our lesson leaned.
– Bi-Weekly Scrum Masters meeting. Moshe.
75% Write a Departemnet Scrum guide. Moshe.
– of the were done
– Run a full cycle of iteration 1 on a PC for evaluation
All lines were reviewed by R&D Manager
• Increase efficiency
– AI: Inquire Why after using Agile the number of bug was not
reduced. 6 weeks. Danny
18

20. R&D Process
Improvement
Other
Root Cause Lessons
needs
Analysis Learned
14 Ch.14 Every Build will have the following 5: NA
artifacts:1. Build in Release management
system2. Should have both production as
well as Debug enabled image3. Release note QC Lead 3 4 4 4 4 4 4 4 3 3 3.70
with CQ list and component details
Project has a software Development plan
that defines the schedule according to the
customer needs 3 2 3 3 3 4 4 4 4 4 3.40
Iteration plan exists 3 3 3 3 3 1 1 1 1 1 2.00
Component Development Plan exists along
with component QC plan 2 2 3 2 1 1 1 1 1 1 1.50
15 Ch.12 a Defects and issues are tracked 2 : somewhat
Configuration Management Plan exists 1 1 1 1 3 1 1 1 1 1 1.20
16 Ch.12 b Defects and issues undergo root-cause 2 : somewhat
analysis
Project has a Resource allocation plan 3 2 3 3 3 2 2 1 1 2 2.20
17 Communication plan is defined 3 : well into it Project has an Integration Plan 3 2 1 4 3 1 1 2 1 1 1.90
Project has a test strategy 3 2 2 3 3 3 3 1 3 3 2.60
18 Ch.13 Artefacts are held within a configuration or 5: NA
Project has all the required documents
document management system managed and reviewed 2 2 1 3 3 4 3 3 3 2.67
19 Ch.15 Project has end of milestone reviews 2 : somewhat
Test cases are prepared, Mapped to
20 Communication about the completion of 2 : somewhat Requirements, reviewed and managed 3 3 2 2 3 3 4 4 3 3 3.00
milestone
APIs have a tool by which they can be tested 2 2 2 2 1 1 1 4 1 1 1.70
Code follows a coding standard 3 1 3 2 2 3 3 1 3 3 2.40
code is reviewed 4 2 4 2 3 3 3 3 3 3 3.00
Self Assessments Self Assessment Analysis
Improvement Plan
19

21. Thank you !
WW_QA
Working Group
20

22. Metrics
• Development progress
– Defects trends
– Code Coverage by tests
– Test Progress
– Requirements Coverage by tests
– Team velocity, Gantt Progress
• Quality prediction
– Counting Escaped Bugs by phases
21

23. Code Coverage Defect trends
Test Progress Static Analysis
22