CACI sought to improve their software code security and reduce vulnerabilities by using qualitative and quantitative techniques. Through interviews and surveys, they identified inconsistencies in their information assurance processes. They then established baselines to quantify vulnerabilities and costs. After implementing improvements based on the qualitative findings, they saw a significant decrease in vulnerabilities and associated costs. CMMI principles helped CACI define objectives, ensure process stability, and trace improvements back to meeting business goals of increased customer satisfaction and competitive advantage.

1. Using Qualitative &
Quantitative Techniques to
Improve Service Delivery
Joanna Patterson
CACI, Operational Excellence
Quality Manager
Maggie Glover
Excellence in Measurement
High Maturity Lead Appraiser

2. Presentation Information
This presentation was presented at the 2015 CMMI
Institute Global Congress in Seattle. The presentation is
from two perspectives.
The first is that of the CACI Quality Manager who led an
effort to improve software code security using qualitative
and quantitative techniques.
The second is that of the high maturity lead appraiser who
facilitated the CACI CMMI for Development Maturity Level
5 appraisal related to this effort.

3. Joanna Patterson
Joanna Patterson is a Quality Manager for CACI
International. Joanna holds a BS in Adult Education, and
an M.B.A. with a minor in Information Security
Management. In addition, Ms. Patterson is in the process
of completing her Doctorate in Information Technology
Management. Ms. Patterson is a recognized Golden Key
Honor Society scholar and completed her doctorate course
work with a 4.0 GPA. Ms. Patterson has over 15 years of
relevant industry experience and has worked on deploying
emerging technologies or large scale network efforts.

4. CACI
Key statistics about CACI International:
• Founded in 1962
• Over 16,000 employees worldwide
• CACI provides information solutions and services in
support of national security missions and government
transformation for Intelligence, Defense, and Federal
Civilian customers.

5. Commitment to Quality
Long standing commitment to quality
• Deploy standards as they are needed
• ISO 9001 – Shipyards, Help Desks, SD
• ISO 20000 – Help Desks, Network Support
• ISO 27001 – Help Desks, Medical, PII, HIPPA
• ISO 28000 – Logistics, Supply Chain
• CMMI for Development ML 3 & 5 – Solutions Development
• CMMI for Services ML 5 – Help Desks, Medical

6. Defining IA Activities
• Information Assurance (IA), for this presentation, is
defined as the activities related to securing the code
developed by CACI for the government.
• The project, specifically the developers, are required to
develop secure code

7. The Challenge
CACI recognized that information
security was, and is, a risk that should
be proactively address.
• Information Security / Cyber-Attacks
• Hackers becoming more
sophisticated
• Increasing government
oversight/regulations

8. The Problem
• It is commonplace to measure the number of defects
found during the software development lifecycle in
hopes of reducing them.
• Information assurance “vulnerabilities” are considered
defects
• An IA defect is a vulnerability which can lead to an
exploit

9. The Problem
• IA defects are most often found when the government
Information Assurance Manager runs a static code scan
• The scan finds potential vulnerabilities based on the
Software Technical Implementation Guide (STIG)
• Example defects: SQL Injection Error, Cross-Site
Request Forgery, Hidden Field, Empty Catch Block

10. Further Explanation
What can you do with a SQL Injection vulnerability?
SQL injection is a code injection
technique, used to attack data-driven
applications, in which malicious SQL
statements are inserted into an entry field
for execution (e.g. to dump the database
contents to the attacker).
Real Headlines!
“Up to 100K Archos customers compromised by SQL Injection
attack”
“Hackers Stole 100K from California ISP using SQL Injection”
“Hackers discloses vulnerabilities in dozens of Military &
Pentagon websites attributed to SQL Injection vulnerability”

11. Doors to Destruction
Government scans found, on average, 3000 errors
Every defect introduced is a potential door for a hacker

12. This is War
Every battle is won before it is fought.
Sun Tzu
Know your weakness
Know your Enemy
Create a battle plan
Change - Adapt

13. Qualitative Research
What is Qualitative Research?
• Exploring issues
• Enhances the reliability of the statistical data
• Increases your knowledge on the topic
• Understand behavior
• Descriptive versus predictive

14. Qualitative Research
Why did CACI use qualitative research methods?
• Constantly changing IA climate
• Need to determine all the variables involved
• Immense influence from outside factors

15. Qualitative Research
What is Qualitative Research?
• Exploring issues
• Enhances the reliability of the statistical data
• Increases your knowledge on the topic
• Understand behavior
• Descriptive versus predictive

16. Qualitative Research
The primary misconception regarding high maturity
efforts is that you can baseline, measure, and build
predictive models without quantifying or qualifying the
problem.
Yes there are IA defects…
Why?
Yes the developers are involved..
How?
Yes the customer has requirements…
What?
Understand your problem before you try and fix it

17. Qualitative Research
Do your research and research
includes talking to your people:
To find out the true origins of
this problem CACI:
• Interviewed the developers
• Surveyed the customer
• Reviewed industry IA
literature

18. Data Collection
• Keep it:
• Standardized
• Open-ended
• Let the people on the front line be heard
• Code the data
• Open Coding – conceptualize the data
• Axial Coding – put it back together, make connections
• Selective Coding – look at the core variables
The following slides give a high level overview of coding
techniques used and should not be considered all inclusive.

19. Interview Question / Open Coding
Question: What is your current IA scanning procedure?
Open Code Actual participant words
Inconsistent procedures The way I do it…
The way the government does it…
Need centralized process
Need communication from government
No training
Government Scans
CACI does not Scan
Retina scans
Gold Disk
The government scans during validation
We perform Retina scans
We do gold disk scans
Lack of Software
Outdated Software
We do not own HP Fortify
Fortify is expensive
Our version is outdated
STIG requirements What is a STIG
Not synced with current STIG template

20. Data Concepts & Categories
I. As data is collected and coded, codes of similar content
appear and allow the data to be grouped into concepts.
II. As concepts emerge, they are grouped to form
categories that are then used to generate a theory.
III. As QA collected data and coded it, several categories of
concepts were identified.
Codes Concepts Categories

21. Concepts & Categories
Open Code Concepts Categories
Inconsistent procedures IA processes inconsistent
Government Scans
CACI does not Scan
Government
Heavy reliance on government intervention
Retina scans
Lack of Software
Outdated Software
Developers
Inconsistent resource utilization
Gold Disk
STIG requirements
Not familiar
Need training
Desire more immediate feedback
Validity of existing errors
Inconsistent and subpar training
STIG requirements
On loan IA person
No IA Support
No external or internal resources
Improper planning or IA defect mitigation

22. Axial Coding Results
The end result are the areas CACI focused on fixing

23. Time to Quantify the data
CACI now moved on to creating baselines in order to
quantitatively manage the IA process areas. Additionally,
there were simultaneous efforts to correct issues in areas
that were not quantifiable (in this instance). For instance,
training, or requirements.

24. Quantitative
Created a baseline of IA defects by category
Initial Baseline:
Mean of 437 CAT I errors
Range:
Over 1000 errors
What this means?
Wide variation in the data with
no outliers means the process is
in statistical control but overall
performance of the process is not
yet predictable.
Note: Just because the process was in statistical control, doesn’t necessarily mean it
was cost efficient or time efficient for resource planning.

25. IA Cost Baseline
Created a IA cost baseline
Average of $27,043 IA cost per month
IA costs incurred to correct
CAT I or CAT II defects
initially only consisted of the
cost of the developer to correct
the code.

26. Made Improvements
Based on the root causes identified during the qualitative
analysis, we identified where opportunities for
improvement existed in the IA process.
• These are identified as the “pain points” of the process.
• These pain points were not specific to one project.
• There were multiple root causes therefore multiple pain
points in the process = multiple changes

27. Decreased Errors
CAT I errors before and after process
improvement (mean decrease from 437 to 57)
Mean: 57
Range:
264
What this means?
The decrease in range and mean
indicate the process is not only
stable but is now in a predictable
state. Projects can anticipate the
number of defects that will need
to be remediated per sprint.

28. Decreased Cost
Average monthly cost decreased from
$27,043 to $16,591; range decreased from
$50,000 to $10,000

29. Predictive Models
Model Type Predictor When to use?
Linear Regression Cost to fix CAT I
defects
No dedicated IA story or hours during
sprint; after adhoc scan performed by
IA Analyst or Government; PM wants
to determine ability of project to meet
ORG objective
Multiple Regression Cost to fix CAT I and
CAT II defects
No dedicated IA story or hours during
sprint; after scan is performed by IA
Analyst or Government; PM wants to
determine ability of project to meet
ORG objective
Linear Regression Hours to fix CAT I
defects
After adhoc scan performed by IA
Analyst or Government

30. The Results!
Higher Customer
Satisfaction
- Delivery of Secure Code
- On Time Releases
- Decreased IA Costs
IA Cost and Predictability
- Answer “How Long” &
“How Much”
- Proactive vs Reactive
Immediate Adoption of IA
Process
- Easy to Adopt and Implement
- Minimal interference with
project management activities

31. Through the eyes of a CMMI High Maturity Lead Appraiser,
Six Sigma Black belt and Scaled Agile Framework (SAFe)
Product Consultant

32. Ms. Glover is a High Maturity Lead Appraiser for CMMI-
Services, Development and Acquisition. She also is an
Intro to CMMI Instructor certified by the CMMI Institute.
Ms. Glover is a former Air Force Captain who served as a
Satellite Office in SPACECMD in Cheyenne Mt. CO. She
has a graduate degree in IT Systems. She is also and ISO
9000, 20000 and 27000 Lead Auditor as well as a Six
Sigma Black belt.
Ms. Glover is currently working at the Cigna Agile Center of
Excellence as a Scaled Agile Framework (SAFe) Product
Manager. She is currently working on developing their
SDLC for their Agile Development Lifecycle.
Margaret Tanner Glover

33. How CMMI HM Helped
• Its all about performance!
• Maturity Level 5 is built on the understanding of the
quantitative measures that are defined in order to lead to
process improvement objectives which enable the
organization to better meet their business objectives and
associated quality and process performance objectives.

34. Remember The Problem?
• It is commonplace to measure the number of defects
found during the software development lifecycle in
hopes of reducing them.
• Information assurance “vulnerabilities” are considered
defects
• An IA defect is a vulnerability which can lead to an
exploit

35. Defining Quantitative Objectives
WHAT DO YOUR CUSTOMERS WANT?
• How do you measure and model your processes to that end
result of customer satisfaction?
We used some Six Sigma objectives:
• The D in DMAIC is Define.
• Define Voice of the Customer. What does the customer want?
• The customer wants defect “free” code as to not have any
vulnerability for exploitation and unethical hacking of their code
• In Agile Development, the Product Owner is responsible for
getting the User Stories from the customer to determine the
requirements of the system.

36. Six Sigma for Voice of the Customer
WHAT DO YOUR CUSTOMERS WANT?

37. Determine Quantitative Objectives
Once you have determined the Voice of the
Customer, then:
What is the process capable of (Determine Voice
of the Process)?
What can be controlled (Controllable Factors)?
• CACI could institute the use of software development tools
for security
• CACI could institute good software security best practices
.

38. Process Stability and Process Capability

39. Determine Quantitative Objectives
Defect free code is the goal. Is it a controllable factor?
• Controllable factor is something in your process that you have
control over. Examples include:
• Training the organization provides
• Coding standards the organization institutes
• Tools that required for estimating
• Encryption and firewalls
• Software Defined Lifecycle (Agile)
• Non controllable factors is something that you do not have
control over. Examples include:
• Weather
• Employees sick time
• Hackers trying to break through your firewall

40. Determine Quantitative Objectives
Defect free code is the goal. Is it a controllable factor?
• Maturity Level 4/ calls for the determination of “Controllable
Factors” which aid in the implementation of Quantitative Project
Management (QPM) especially the following practices:
• QPM SP 1.4 Select measures and analytical techniques to be
used in quantitative management.
• QPM SP 2.3 Perform root cause analysis of selected issues to
address deficiencies in achieving the work’s quality and
process performance objectives.
.

41. Determine Quantitative Objectives
• CMMI helped define the Quantitative Process
Performance Objectives which traced to the VOC and
hence CACI’s business objectives (OPP).
• Can you then satisfy your customer with the current
process?
• One more requirement:
• A Stable process has to be in place which can be
measured by the institutionalization of ML3 practices.
• Before the process can be “Capable” it has to be “Stable”.

42. Determine Quantitative Objectives

43. Determine Quantitative Objectives
• One more requirement:
• A process cannot be released to production until it has
been proven to be stable.
• We cannot begin to talk about process capability until we
have demonstrated stability in our process.
• A process is said to be stable when all of the response
parameters that we use to measure the process have both
constant means and constant variances over time, and also
have a constant distribution.
• This is equivalent to our earlier definition of controlled
variation or a Stable Process.

44. Quantitative Management for ML4
44
Process Area: OPP
The IA Process was stable and predictable
Now What?
First – CACI listened to the “voice of the customer”. CACI
created models based on project management feedback and
customer input
Second – CACI listened to the “voice of management”. Models
were created to predict the projects ability to meet the org measure

45. Determine Quantitative Business Objectives
Business Objectives at CACI are to increase customer
satisfaction
• Maturity Level 5 required the organization to proactively
manage the organization's performance to meet its business
objectives.
• Using Organizational Performance Management (OPP)
• OPP SP 1.1 Maintain business objectives based on an
understanding of business strategies and actual performance
results.
.

46. Traceability
46
BusinessObjective
Increase business for the division by achieving a X% success rate on re-
competes, and X% success rate on new bids by June 30, 2014 (for all
performance based awards).
SubGoals
Exceed Customer Expectations
Maintain a competitive advantage through
quality and process improvement
Reduce physical and
information system
weaknesses
CMMI requires the linkage of the business objectives to the QPPO
(Goal provided is an example for the purposes of confidentiality)
Process Area: OPP

47. How CMMI HM Helped
• When CACI determined the quantitative objectives of the
IA efforts, they were able to determine the baselines and
models and ensure the business objectives were being
satisfied.
• CMMI Requirements for ML5 in Causal Analysis and
Resolution (CAR) was used to determine where to
implement the selected action proposals and evaluate the
effect of those implement actions.
• Example: “using the root causes identified during
qualitative analysis, we identified where opportunities for
improvement existed in the IA process”

48. How CMMI HM Helped
• The requirements at ML5, asking an organization to use
their business performance and manage it using statistical
techniques which lead to identify potential areas for
improvement that could contribute to meeting those
objectives were realized when the physical and
information system weaknesses were found and
eliminated.
• This led to a decrease in dollars saved, allowing CACI to
meet their business objective of “Exceeding Customer
Expectations”.

49. 49OPM
Questions?
You have questions?
We have answers!