1. STRAYER UNIVERSITY
A DIRECTED STUDY PROJECT SUBMITTED TO THE
FACULTY OF THE GRADUATE SCHOOL OF BUSINESS
IN CANDIDACY FOR THE DEGREE OF MASTER OF
SCIENCE INFORMATION SYSTEMS
DISTANCE LEARNING CAMPUS
SCOTT R. STADELHOFER
MARCH 2003

2. ABSTRACT
Since the early 1990s, organizations have been improving their software processes
according to the requirements stated in the Software Engineering Institute’s (SEI)
Software Capability Maturity Model (SW-CMM). During that time, many Science
Applications International Corporation (SAIC) organizations have increased in their level
of software maturity. Many organizations have achieved SW-CMM Level 3, and some
have achieved Level 2. However, very few organizations have achieved maturity levels
at or above Level 4. SEI Level 4 represents a very important level of maturity because
this is where project processes truly become stable. This stability is accomplished
through the use of quantitative and statistical management techniques such as run charts,
control charts, confidence intervals, prediction intervals, and test of hypotheses. Using
these techniques, the data from statistical management helps the project predict whether it
will be able to achieve its quality and process performance objectives quantitatively. For
example, with a Level 4 process in place, a quantitative determination can be made if
schedule corrections are needed. Also, the quality of the software using defect data from
peer reviews and tests can be predicted for software that needs to conform to very high
quality standards. Before achieving Level 4, process performance is mostly judged
intuitively. The competitive environment requires that organizations strive to achieve
Level 4. A measured software maturity level, as a result of a CMM Integration (CMMI)
Based Appraisal for Internal Process Improvement or Software Capability Evaluation,
can be used as a discriminator for winning contracts. An organization’s Level 4 status
makes a significant difference in procurements in the avionics, information system, and
e-commerce arenas.
This study begins with an overview of the Capability Maturity Model Integration
(CMMI) process improvement structure as a context for studying metrics techniques.
Chapter one also includes a statement of the problem and its significance. The objectives
of this study on metrics techniques and the research methods are also explained in
Chapter one. Chapter two includes the bulk of the research on metrics techniques and
econometrics. Interviews with experts in the metrics field enlighten readers with real
world experiences in Chapter three. This study includes accounts from five interviews
with metrics experts that work for Science Applications International Corporation on
various projects and their various metrics techniques and opinions on CMMI. Chapter
four details the summary and conclusions of the research on metrics by digesting the
Cost-Benefit Analysis Method (CBAM) as well as other methods. The economic value
of metrics techniques is an often-neglected issue in dealing with CMMI. The
Architecture Tradeoff Analysis Method (ATAM) uncovers the architectural decisions
that are made for a system and links these decisions to business goals and quality
attributes. The CBAM builds on this foundation by enabling engineers to determine the
costs and benefits associated with these decisions.
ii

3. Dedication
To my wife and bride Carmen
iii

4. TABLE OF CONTENTS
CHAPTER PAGE
ABSTRACT........................................................................................................................ii
Dedication...........................................................................................................................iii
CHAPTER 1 – INTRODUCTION......................................................................................1
INTRODUCTION.........................................................................................................1
Context of the Problem..................................................................................................1
Statement of the Problem...............................................................................................2
Significance of the Study...............................................................................................5
Objectives of the Study..................................................................................................7
Research Methods........................................................................................................10
CHAPTER 2................................................................................................................12
Metrics In Today’s Environment.................................................................................12
Metrics and Software Process Management................................................................13
Integrating Metrics with the Software Process............................................................15
Applying Metrics to Process Management..................................................................18
Examining Metric Results............................................................................................21
Structured Processes and Data Reliability...................................................................23
Measurement for CMMI Implementation....................................................................23
An Effective Metrics Process Model...........................................................................27
Applying the Model - Establish and Validate Organizational Goals...........................29
Creating a Metrics Plan................................................................................................30
Reviewing the Metrics Plan.........................................................................................31
The Architecture Tradeoff Analysis Method (ATAM)...............................................34
The Cost Benefit Analysis Method (CBAM)..............................................................35
iv

5. Incorporating Uncertainty............................................................................................39
Inter-Scenario Significance..........................................................................................44
Architectural Strategies................................................................................................44
Side Effects..................................................................................................................44
Determining Benefit and Normalization......................................................................45
Cost..............................................................................................................................46
Calculating ROI...........................................................................................................46
Dealing with Uncertainty.............................................................................................47
Quantitative Economic Models to Qualitative Engineering Judgments......................48
Portfolio Analysis and the Security Technology Selection Problem...........................49
Security Technology Selection as Portfolio Analysis..................................................50
Mismatch Between Model and Available Data...........................................................51
Addressing the Mismatch............................................................................................51
Converting Qualitative Information to Quantitative Ranges.......................................52
Analysis Based on Partial Orderings...........................................................................53
Establish Validity of Using Quantitative Techniques after Scale Transformations....53
Defect Potential in Assessing the Economic Value of Process Improvements...........55
Defect Potential and Its Impact on Software Development Organizations.................56
A Model of the Cost of Defect Potential and Information Needs................................58
Technology, Costs, and Defect Potential.....................................................................59
COCOMO II................................................................................................................61
COCOMO and Function Points...................................................................................63
COQUALMO..............................................................................................................63
CORADMO.................................................................................................................65
v

6. CORADMO Description.............................................................................................65
COPROMO..................................................................................................................66
COPSEMO...................................................................................................................66
COSYSMO..................................................................................................................66
Code Count Tools........................................................................................................67
CHAPTER 3................................................................................................................69
Interviews with Five SAIC Employees........................................................................69
Interview 1 – Robert Luschenat and Wayne Dow.......................................................69
Interview 2 – Pamela Bank..........................................................................................74
Interview 3 – Frances Soskins.....................................................................................83
Interview 4 – Janice Rouiller.......................................................................................90
Interview 5 – Pamela Schoppert..................................................................................97
CHAPTER 4..............................................................................................................104
SUMMARY AND CONCLUSIONS........................................................................104
APPENDIX A............................................................................................................113
BIBLIOGRAPHY......................................................................................................117
vi

7. List of Tables and Figures
Figure 1 – Measurement Activities and Their Relationship to the Key Responsibilities of
Process Management...................................................................................................14
Figure 2 – Context of the CBAM................................................................................37
Figure 3 - Process Flow Diagram for the CBAM........................................................41
Figure 4 - Some Sample Utility-Response Curves......................................................42
Figure 5 - COCOMO Black Box Overview................................................................63
Table 2 - Expanded Measurements for CMMI Maturity Level 3..............................115
Table 2 - Expanded Measurements for CMMI Maturity Level 3..............................116
vii

8. CHAPTER 1
INTRODUCTION
Context of the Problem
While each business organization has a diverse customer base, the common set of
goals for software products and services are: timely delivery, affordability, high quality,
and satisfied customer base.1
Measurements tied to these goals can also be used to
evaluate infrastructure and project-management activities and to calibrate system
engineering and software development capabilities.2
All software projects require the use of metrics to effectively manage the
projects. Quality software projects follow the highest possible level of the Capability
Maturity Model Integration (CMMI). At Level 1, project managers and development
staff do not plan the activities or collect metrics that would allow them to control their
project. There is little or no control of processes. Level 2 requirements are primarily
focused on maturing program management related processes (requirements
management, project planning, and project tracking). Once management practices are
instituted within the organization, the process is considered to be repeatable. At Level
3, a standard and consistent process for use on all projects within the organization is
defined. This process includes engineering practices and can be tailored for each
project depending on the project’s characteristics. At Level 4, standard processes
become consistent and predictable on projects. This is accomplished through the use of
1
Putnam, Myers, Measures for Excellence: Reliable Software On Time, Within Budget, Yourdon Press,
Englewood Cliffs, NJ, 1992.
2
Ibid.
3
Edward F. Weller, Using Metrics to Manage Software Projects, [article on-line], available from
http://www.stt.com/ieee1994.pdf, accessed 22 February 2002.
1

9. quantitative objectives and measurements. This is called the managed level. Finally, at
Level 5, processes are continually improved using quantitative objectives and
measurements for the entire organization. This is called the optimizing level.3
Statement of the Problem
Foremost in the quest for good metrics is a definition of a “good metric.”
Although a review of current literature on metrics indicates many definitions of the term,
they could be summarized as a measurement that helps the right level of leadership make
the right decisions in a timely manner, based on fact rather than “gut feeling.” With that
definition in mind, the majority of authors agree that the first step in good metrics
collection is understanding the goal. Rather than ask what is to be measured, ask what is
important to the organization and its customers. Organizations must validate
organizational goals and objectives with their customers, suppliers, and senior managers
when they publish a strategic plan. Re-validated semiannually, this document can outline
the direction that is expected in the next few years. Sometimes missing from an
organization’s strategic plan is a link of metrics to measure the progress of these goals.
In re-validating this strategic plan, using metrics as a tool to measure these goals, many
agree that the goals can be too general because they can not be measured. These goals
and objectives have to be revaluated, ensuring that each objective has an associated
measurement to ensure progress. Although these goals are important to every
organization, it can be difficult to focus on defining clear and measurable goals based on
2

10. what is important to customers. Senior management can be skeptical about the value of
spending time defining such goals.4
Most disciplines that succeed have created a linkage between success, control,
and measurement. Although measurement has been in existence for many hundreds of
years, it has only been in the last few years that it has been recognized as necessary in the
software industry. Even with this recognized need, all levels of managers often
misunderstand the need and knowledge of software measurements. Management chooses
success or failure by the way it uses (or abuses) management information. What many
managers fail to realize is that management is responsible for the proper mix of project
attributes that will increase the factors needed for success. Project attributes that
influence all software processes include:
• People - What skill sets (including the user) do they have?
• Tools - What tools (e.g.; platform, language, code generator) are provided?
• Techniques - Is a methodology followed? Does the project team use Joint
Application Development (JAD) and/or prototyping sessions? Are there code walk-
throughs?
• Environment - Is the work place conducive and does it contribute to the development
process?5
The specific goals of software metrics are the following:
• Provide early-warning and in-process indicators to Software and Project Managers to
allow early, sound adjustments for meeting schedule and cost commitments
4
Capt. Thomas Augustine and Charles Schroeder, An Effective Metrics Process Model, CrossTalk, 23
February 2001, [journal on-line]; available from http://www.stsc.hill.af.mil/crosstalk/1999/06/
augustine.asp; accessed on 5 March 2002.
5
John Van Orden CFPS, Consulting Director, Who Needs Metrics, Part 2, 2001 Quality Plus Technologies
and John Van Orden, available from http://www.qualityplustech.com/; accessed 12 August 2002.
3

11. • Accurately estimate development size and effort for new proposed work as well as
enhancements to existing software
• Provide basis of estimates to customers and management
• Provide a common statistical basis for understanding and assessing the software cost,
schedule, productivity, and quality of any project
• Measure software process improvement
• Provide a basis for comparison with industry.
Most software development organizations use software metrics to some extent.
Overall software project goals are to have predictable performance; to reduce cost,
schedule, and technical risk; and to increase product quality and customer satisfaction.
The motivation to use metrics stems from the desire to have early warning indicators that
help manage software projects more effectively, assist the developers to make better
software engineering decisions, to become better estimators of cost and schedule, and
help establish metrics repositories to support software estimates and productivity
projections. Customers motivate development organizations to use metrics by using the
SEI CMMI or equivalent methodologies during source selections and by making metrics
a contractual requirement.
Metrics have an essential role in establishing a project’s target effort, cost, and
schedule and in gauging its progress toward these targets. Metrics are related to cost
estimation in three areas: project tracking, replanning, and calibration of estimates and
estimating models. Recognition of these links is essential for project success:
• In-process measures of effort, cost, schedule, and technical activities collected during
the project enable the Project Manager to accurately assess a project’s progress.
Significant deviations of the actual values from the planned values trigger detailed
4

12. analysis of project performance, which should lead to actions taken early enough to
avoid subsequent schedule and cost impacts.
• Actual measured values for project parameters help reestimate project effort, cost,
and schedule during the course of the project. This is done whenever project
parameters change and/or unexpected events occur. (For example, new requirements
are often added or discovered.) Such reestimates provide more accurate estimates for
use in better managing the rest of the project.
• Cost estimation provides planned values for project effort, cost, and schedule. Actual
measured values from completed and in-process projects, combined with the
measured size, are used to calibrate various estimating models for effort, cost and
schedule, leading to more accurate estimates for future modifications and new
projects.6
Significance of the Study
This study examines issues to consider in evaluating tools and making metrics
tool selections. It mainly analyzes the tools that are available today and how they
compare to each other. The conclusions and recommendations based on these findings
concentrate on the realized value of using metrics to the organization by examining
econometric issues such as the Cost Benefit Analysis Method.7
Based on the maturity of
a standardized approach to metrics, the need for a universal metrics tool is fulfilled with
the integration of niche tools. A practical way to evaluate how well a niche tool will
work in an automated environment is how well the data can be used by other tools in the
environment. Although it is not expected that any universal metrics tools will be found in
the near future, several tools support all of the core attributes to varying degrees. The
6
SAIC Metrics Handbook & Collection Guide, Metrics Working Group, ISSAIC; Intranet; accessed 8
January 2002.
7
Giles, Alan E., and Gregory T. Daich, Metrics Tools, CrossTalk, February 1995,
http://www.stsc.hill.af.mil/crosstalk/1995/02/, accessed 5 March 2002; Internet.
8
Gregory T. Daich, Alan E. Giles, Universal Metrics Tools, Software Technology Support Center,
available from http://www.stsc.hill.af.mil/crosstalk/1995/09/universa.asp, accessed 5 March 2002.
5

13. classification of these tools as primary software metrics tool sets seems to be a useful
distinction.8
Even if a metrics plan is perfectly implemented, it is still incomplete unless the
correct level of management makes decisions based on the collected metrics. It has been
well-documented that management buy-in and commitment are vital before a metrics
process can work. Organizations must ensure that senior management understands the
implications of the metric analysis through monthly metrics meetings with senior
managers, midlevel managers, and those who collect and analyze the data. This high
visibility is crucial for a successful metrics program. Without definite due-dates and
justification for information collection and analysis, senior managers likely would not set
metrics as a priority. Everyone who collects, analyzes, or makes decisions based on
metrics data has to be aware of the process, due-dates, and most important, that the
metrics are being used to make corporate decisions. When all parties involved
understand the importance of these metrics, they are likely to make an effort to collect
accurate data, analyze it, and ensure reporting is done quickly to aid in the decision-
making process. To be effective, even the most perfect plan needs constant review. A
successful metrics program guarantees that data is collected and analyzed consistently,
and most important, it ensures that the appropriate people are making timely decisions
based on fact. All that remains is a semiannual review to ensure the organization stays on
track with the decisions it is making based on these metrics. The literature encourages an
analytical view of metrics, thinking not of individual measurements but of a system that
produces good organizational or corporate decisions. This process has been proven
8
6

14. accurate throughout the available literature and in practice. All organizations could
benefit from implementing a structured metrics program.9
Objectives of the Study
Traditionally, the study of software engineering has been a primarily technical
endeavor with little attention given to economic benefit. Design and implementation
methods are based on technical merits instead of the truly economic ones. Students are
taught decision-making skills only in a technical context. This is in drastic contrast to the
reality of software engineering, which is always a commercially motivated activity.
Developed software products add value to a firm. The costs of development are intended
to be outweighed by the returns from the product in order to gain a profit. Dollars spent
in one activity do not magically get reallocated to another. Today’s software engineering
research has failed to completely address the econometric needs of the software
engineering industry. This shortfall of relevant econometric research creates many
problems. More and more, software engineers in organizations whose very existence is
dependent upon the profitability of their software are finding themselves poorly equipped
to make metrics-related and thus economic-based decisions.10
Earned Value Management (EVM) is a method for integrating work scope,
schedule, and budget and for measuring project performance. It compares the amount of
9
Capt. Thomas Augustine and Charles Schroeder, An Effective Metrics Process Model, CrossTalk, 23
February 2001, [journal on-line]; available from http://www.stsc.hill.af.mil/crosstalk/1999/06/
augustine.asp; accessed on 5 March 2002.
10
Warren Harrison, Hakan Erdogmus, and Rick Kazman, The Fourth International Workshop on
Economics-Driven Software Engineering Research (EDSER-4), Portland State University, National
Research Council of Canada, Software Engineering Institute, available from
http://www.cs.virginia.edu/~sullivan/edser3/, accessed 20 February 2002; Internet.
7

15. work that was planned with what was actually accomplished to determine if cost and
schedule performance were achieved as planned. For organizations using Earned Value
Management or that plan to implement EVM during Capability Maturity Model
Integration (CMMI) implementation, this study attempts to provide guidance for cost-
effective process improvement and appraisal. Mapping and comparison tables between
CMMI and the U.S. national standard on EVM are explained. These tables can be used
to identify practices within CMMI that are not included in the EVM standard but, if
added to an organization’s processes, will strengthen adherence to EVM principles.
The basic principles of an EVM system typically include the following:
• Break down the program work scope into finite pieces, called work packages that can
be assigned to a responsible person or organization for control of technical, schedule,
and cost objectives.
• Integrate program work scope, schedule, and cost objectives into a performance
measurement baseline against which accomplishments can be measured.
• Objectively assess accomplishments at the work package level.
The following key process areas (KPAs) are those that are mainly related to Earned
Value Management Systems (EVMS):
• Measurement and Analysis
• Project Planning
• Project Monitoring and Control
• Requirements Development
• Requirements Management
• Integrated Project Management
8

16. EVMS also relates to specific practices in the following process areas:
• Supplier Agreement Management
• Risk Management
• Process and Product Quality Assurance
Each of these process areas plays an important part in specifying and tracking
project work, developing operational definitions of how that work is measured, or
verifying that completion criteria for work products have been satisfied.
The U. S. national standard for EVM is Earned Value Management Systems
(EVMS). U. S. government policies for performance-based acquisition management
always require the use of performance-based management systems that meet the
guidelines of EVMS. The Office of Management and Budget also requires that all
agencies of the Executive Branch of the government that are subject to Executive Branch
review must use performance-based management systems that meet the EVMS
standard.11
Despite the criticality of software architectures to successful systems, software
architects do not have well-developed decision making structures to reason about the
costs and benefits of the various design options that are available. The Architecture
Tradeoff Analysis Method (ATAM) provides a framework for software architects to
analyze the technical tradeoffs involved in designing and maintaining a software system.
But these technical tradeoffs have basically ignored economic considerations. The
method for doing economic analyses of software and systems that centers on
11
Paul Solomon, Using CMMI to Improve Earned Value Management, Software Engineering Process
Management, October 2002 [website]; available from http://www.sei.cmu.edu/pub/documents/02.reports/
pdf/02tn016.pdf; Internet; accessed 11 November 2002.
9

17. architectures is called the Cost Benefit Analysis Method (CBAM). The CBAM builds
upon the ATAM adding metric techniques that help the analyst to better understand the
benefit and cost implications of the architectural design decisions being made. Given the
fact that there are large uncertainties, both technical and economic, in the architecture
design stage of a software system, architects face risks with a system’s ability to achieve
the business goals. One of the advantages of this approach is the fact that it forces the
stakeholders to quantify the risks by explicitly quantifying the benefits of architectural
decisions, costs, quality attribute responses, and the uncertainty surrounding the metric
estimates.12
Research Methods
The research for this study will be accomplished using various sources available
on the World Wide Web. One of the primary sources is the website for the Software
Engineering Institute at Carnegie Mellon University. The Software Engineering Institute
is a federally funded research and development center sponsored by the U.S. Department
of Defense. Other sources include articles from the Software Technology Support
Center’s publication called CrossTalk. The original research portion in Chapter 3 will be
accomplished by collecting five interviews with experts in the field of metrics from
different locations at Science Applications International Corporation (SAIC). These
interviews collect opinions on various metrics techniques used by different groups at
SAIC. Chapter 4 will serve as a conclusion and will focus on the economic value of
12
Jayatirtha Asundi, Rick Kazman, and Mark Klein, Using Economic Considerations to Choose Among
Architecture Design Alternatives, [article on-line], available from http://www.sei.cmu.edu/publications/
documents/01.reports/01tr035.html, (Software Engineering Institute, 2001, accessed 15 February 2002),
identifier no. CMU/SEI-2001-TR-035.
10

18. metrics techniques by examining Earned Value Management (EVM), Cost Benefit
Analysis Method (CBAM), and Architecture Tradeoff Analysis Method (ATAM).
11

19. CHAPTER 2
Metrics In Today’s Environment
Software measurement alone cannot solve the problems of new technologies,
increasingly competitive markets, intense competition for scarce experienced personnel, and
the demands for faster responsiveness. Developers also deal with the problems of open-
ended requirements, uncontrolled changes, inadequate testing, insufficient training, arbitrary
scheduling, lack of funding, and standards issues such as product reliability and suitability.
Metrics techniques can clarify and focus the understanding of these issues and problems.
Sequential metrics of quality attributes for products and processes allows for initiating and
managing process improvement activities. Astute software organizations are able to predict
and commit based on their software products. Efficient metrics techniques help these
organizations understand the needed capabilities so that production and delivery plans can be
made to develop products and services. Metrics aid in detecting trends and anticipating
problems, which produces better cost control, risk reduction, improved quality, and achieved
business goals.
Process management guarantees that the procedures in an organization are performing
as expected. It ensures that defined processes are followed and makes process improvements
to meet business objectives. The software processes that organizations use to develop
software products are critical to the execution of strategies and plans to obtain these
objectives and goals. Controlling these processes helps the organization predict product and
12

20. service characteristics, estimate costs and schedules, and improve the effectiveness,
efficiency and profitability of corporate and technical operations. Process performance is
quantified by measuring product attributes directly. The following figure displays common
business goals that correspond to project and process issues as well as attribute examples that
can be measured to judge the performance of a process in relation to these issues.
Business Goals Project Issues Process Issues Measurable Product and
Process Attributes
Increase Function Product Growth
Product Stability
Product
Conformance
Number of Requirements
Product Size
Product Complexity
Rates of Change
% nonconforming
Reduce Cost Budgets
Expenditure Rates
Efficiency
Productivity
Rework
Product Size
Product Complexity
Effort
Number of Changes
Requirements Stability
Reduce the Time to
Market
Schedule
Progress
Production Rate
Responsiveness
Elapsed time, normalized
for Product Characteristics
Improve Product
Quality
Product
Performance
Product Correctness
Product Reliability
Predictability
Problem
Recognition
Root Cause
Analysis
Number of Defects
Introduced
Effectiveness of Defect
Detection Activities
Table 1 – Business Goals, Project and Process Issues, and Related Measurable
Attributes
Metrics and Software Process Management
There is a relationship between process management and the planning and
application activities of process measurement. The following figure illustrates the
13

21. interactions between measurement and process management as summarized in the text
below.
Figure 1 – Measurement Activities and Their Relationship to the Key
Responsibilities of Process Management
Define the Process – A process is an organized combination of workers, materials,
energy, equipment, and procedures engaged in producing a certain end result such as a
product or service. Before selecting and implementing metrics, each contributing
element of the process must be identified. A thorough understanding of the process
operation and objectives must be obtained by those involved in process management.
Plan the Measures – Metrics planning is based on knowing the defined software
process. Product, process, and resource-related issues and attributes are identified.
Process quality and product measures are selected and defined. The necessary items for
collecting and using the measurements to assess and track process performance are
integrated into the software process.
14
Customer Requirements and Business
Needs
Improve
Process
Process Improvements
Define Process
Plan Measures
Execute
Software
Process
Process
Definiti
People,
Resources,
Work Products
Control
Process
Process
Measurements
Products
Apply
Measures
Product
Measurements
Process
Performance

22. Execute the Software Process – Processes are executed by the software development
organization. The product, process, and resource attributes that were identified are
measured during and at the end of each software process.
Apply Measures – Applying metrics uses the measurements obtained while executing
the software process. Software process and product data produced by the process are
collected, saved, and analyzed so it can be used to control and improve the process.
Control the Process – If product and performance measurements indicate that the
process varies in unexpected ways, then actions should be taken to eliminate assignable
causes, stabilize variability, and if needed return the process to its natural level of
performance.
Improve the Process – When metrics indicate process variability comes from a constant
system of chance causes, process performance data can be used to guide efforts to change
the level of performance. Improvements whose benefits are later validated by
measurement can be used to update and evolve the process definition.
Integrating Metrics with the Software Process
After identifying process issues and selecting and defining measures, the third
measurement planning activity is integration with the software process. This involves
analysis, diagnosis, and action. Analysis means identifying the metrics that the
organization collects now and how they are utilized. Diagnosis is evaluating the extent to
which these metrics can be useful in meeting identified needs and determining where
more work is necessary. Action is directed at translating the diagnosis and analysis
results into action plans for collecting and using more data.
15

23. Analysis sets the baseline for later work. Knowing which metrics are collected
and how they are used gives a starting point for implementing defined measures.
Measurement activities should already be in place. New techniques can be met with
resistance sometimes. Most organizations build on things, which are currently in use.
This will strengthen and refocus them in the process. When analyzing metrics
techniques, five questions should be answered:
• What data elements are needed for the measures?
• Which ones are currently collected?
• How are they collected?
• What processes provide the data?
• How are data elements stored and recorded?
There are more potential data sources than are apparent at first. Mental models for the
processes can help locate the sources. Many sources can exist for data on defects and
problems. All of these sources result in analyses and corrective action. These results
along with status information are stored in a database.
Diagnosis evaluates the collected data elements and decides how well they meet
the requirements of goal-driven measures by proposing actions for using the data,
adapting the data to needs, adapting needs to the data, and obtaining whatever is missing.
Diagnosis is evaluative and judgmental when analysis is fact-finding. Diagnosing
identifies alternatives and prepares for finding solutions. Questions asked include:
• What existing processes and metrics can be used to satisfy data requirements?
16

24. • What elements of measurement definitions or practices must be changed or
modified?
• What new or additional processes are necessary?
Action translates the results of analyses and diagnoses into implementable
operations. Actions arrive at solutions and make them happen. It involves identifying
tasks, allocating resources and responsibilities, and following up to guarantee that the
actions actually take place. Action begins with identifying the elements that are built on
or addressed in the metrics plan. Things that need to be done before writing the plans
are:
• Identify sources of data within existing software processes.
• Define methods that can be used to collect and report the data.
• Specify tools required to collect, report, and store data.
• Determine requirements for points in time and frequencies of measurement.
• Document data collection procedures in detail.
• Determine personnel who will use the data.
• Decide how the data will be analyzed and reported.
• Prepare a data definition and collection process guide.
Data storage and access requirements need to be analyzed. This means identifying or
deciding:
• Historical retention needs
• Personnel to collect, store, maintain, and access the data
17

25. • Organizational levels to be served since serving more than one level indicates
a need for more than one database
• Granularity of the data (Will it be retained as recorded or aggregated in some
way?)
• Procedures to be used for dynamically editing and verifying data as data
elements are entered into the database
• The number of individuals with access to the data
• The need for recording the definitions associated with the data so that users
can tie it to the descriptive information needed to correctly use the data.
Also, close attention should be paid to data privacy and security. This is very important
for data that is used to evaluate individual or team performance. A large amount of
evidence shows that the best way to make measurement fail is to let people think it will
be used against them.13
Applying Metrics to Process Management
Two things to look at when trying to improve process performance are process
activities and subprocesses; and things used within the process that originate outside the
process. Process knowledge will guide in directing attention inward or outward. Both
inward and outward attention may be needed at the same time. When looking inward the
strategy will be to decompose or divide and conquer. When a process is broken down
into its component activities and subprocesses, points where intermediate products are
produced will be found. The output streams from these processes are candidates for
measurement and analysis. Control charts can be used if reducing variability is an
objective. When looking outward, the material, resource, and guideline characteristics
13
William A. Florac, Robert E. Park, and Anita D. Carleton, Practical Software Measurement: Measuring
for Process Management and Improvement, [article on-line], (Software Engineering Institute, 1997,
accessed 19 December 2002), Guidebook, identifier no. CMU/SEI-97-HB-003.
18

26. can limit what processes achieve. If materials, resources, and guidelines inhibit the
improved performance, the organization may have to go back to these entities’ origins to
find ways to change performance levels or reduce variation in process results. There is a
whole host of process entities whose origins are outside the process. They range from
people, time, and money to policies, regulations, and instructions. These things need to
be looked at to improve a process. If attribute stability and variability connected with
these entities can influence the process operation, then implementing measures that
quantify, control, and improve those attributes will need to be considered.
In looking upstream, consider a system testing process. One of the best ways to
cut the time, cost, and rework of system testing may not be in the testing process but in
the incoming artifact quality that it tests. Improving on the economics of testing is not
possible without improving important parts of the upstream processes. These are the
activities where activities are inserted and the points where prevention, detection, and
removal activities are implemented and made more effective.
When analyzing process performance, they are affected by outside elements.
Dictated schedules and tight financial situations are two typical examples. Restricted
time and money can cause organizations to cut back on the efforts of preventing and
detecting defects in design and coding. Upstream processes will probably meet preset
time and budget targets, but the downstream results may outweigh the upstream benefits.
Looking beyond the testing process means balancing the overall software process
performance. Or if the organization knows how well inspections work, has the data to
19

27. suggest the optimal inspection levels, and would normally perform at that level, then this
is just an example of process noncompliance.
When changing a process in order to improve it, the organization must analyze the
effects the change will have on other processes in the workflow. The Law of Unintended
Consequences is unavoidable. Downstream effects are changes that reduce variability
but keep the same average. They improve downstream activity efficiency. Reduced
variability rarely has negative effects. However, changes that move the average do affect
downstream processes and must be monitored. It is only recommended to make these
changes if the new level benefits the people and processes that depend on the results of
the improved process. Upstream effects are those in which the feedback loops are
measured output qualities from a process that are used to control or specify its input
attributes. So high defect rates found in testing can cause increased inspection time,
which changes output quality from earlier stages of the process. The knowledge of and
quantifying the reduced variability benefits or new performance levels can significantly
aid in winning management support and funding for researching methods to move the
process mean and for making process changes that reduce variation or adjust its
performance level.14
Development or process managers can only measure products, processes, and
resources that are being used by a project. A developer’s software process indicates
which attributes can be measured and how these measurements can be made. When
setting up an overall measurement process, the developer’s proposed and actual software
processes must be considered. There may be major differences between the desired or
14
Ibid, pp. 124-126.
20

28. envisioned software processes and those that get documented, interpreted, or executed.
The differences result from deficiencies in process description, comprehension,
traceability, and interpretation. This shows the need to fully understand the software
process (and the factors that affect the process) which is in direct proportion to the
measurement requirement detail. This knowledge is vital during the metric process
planning and analysis stages.
Examining Metric Results
Metric data analysis and interpretation must be handled in the context of the
processes and environments that produce them. Metric data alone is neither good news
or bad news. For example, a report that shows zero defects in the first two months after
product release may be taken as very good news if the product has a large customer base.
Or it could be very bad news if there are very few customers using the product. Metrics
results have to be interpreted in the context of other information on the process or product
to decide if action is necessary and what action to take. Unanticipated measurement
results usually require further information to correctly assess the meaning of the
measurement.
If a picture paints a thousand words, then measurement for software process
management and improvement as a valid metric is worth a thousand guesses. Well-
supported, objective quantitative data and facts overcome unsubstantiated and subjective
opinions. Clearly understandable factual data facilitates accurate analysis and ensures
agreement on what is happening and what ought to be happening in a software process.
Software metrics allow for clear and objective communication within the software
organization as well as with customers.
21

29. Most software process management and improvement data results from individual
software projects or activities within them. This data is specific to the particular process
being used in terms of the definition and the context in which the metric results are to be
interpreted. Because process management involves collecting, analyzing, and comparing
data from multiple projects and products, great care should be exercised when
aggregating data as well as when comparing projects or products, plotting trends, or
analyzing the benefits of process improvements. When comparing or aggregating data,
the implied assumption is that all other things are equal. Differences between processes
can invalidate the analysis or require analytical changes in the method of data
aggregation and collection. These issues can be eliminated or reduced by using common
data definitions from projects that use the same processes and measurement techniques.
This may not be possible if the analysis level is more detailed than the commonality level
permits. When comparable data are not directly attainable, mathematical modeling is
used to achieve normalized measured results. If models are used to account for different
factors, it is valid to compare results obtained across several projects over time. The best
mathematical models are those which have been validated and well-documented with
empirical evidence. These models must account for the contextual data that documents
the similarities and differences in the processes and environments where the original data
was collected. Process management requires the collection of additional data over and
above what is usually necessary for project management. Its purpose is to enable valid
normalizations, comparisons, aggregations, and analyses to be made.
22

30. Structured Processes and Data Reliability
Quality process management uses measurements of processes, products, and
resources that are based on disciplined and controlled metrics techniques. A good metric
process has not only clearly defined data but also operational methods for relating
process issues to the required metrics and a defined process for collecting and analyzing
measurement results. Making efficient decisions to control, stabilize, predict, and
improve data reliability and software performance requires a strong confidence in the
data. Therefore, metric data must actually represent the process, relate to the issues at
hand, and be analyzed with techniques that are consistent with the nature of the data and
its environment.15
Measurement for CMMI Implementation
The core set of measurements used to monitor and track project status and to
determine if organizations are satisfying their respective business goals are effort, cost,
schedule, and customer satisfaction. While each organization has a different customer
base, the same set of goals for the products and services are:
• Timely delivery
• Affordability
• High quality and a satisfied customer base.
Measurements connected to these goals are also used to evaluate infrastructure
and project management activities and to calibrate system engineering and software
development capabilities. Specific metrics are selected and tailored for a specific
organization and project based upon informational needs and economic practicalities.
15
Ibid, pp. 188 – 192.
23

31. The organization makes a solid business case to justify the cost of collection, analysis,
and reporting of the selected measures. The core metrics are intended to give insights
into project health and probability of success and to help managers manage development
activities more effectively. They also support the organization in assessing how well
business goals are being met. This information is be useful to the following types of
personnel:
• Measurement practitioners that plan and implement measurement processes
• Line managers, project managers, software managers, systems engineers, and
engineering managers
• Test managers, requirements managers, developers, and others involved in
providing quality products and services to SAIC customers on time and within
budget.
Companies are motivated to use measurements to provide early warning
indicators that help them manage projects more effectively, assist engineers and
developers to make better engineering decisions, help managers become better estimators
of cost and schedule, and establish measurement repositories to support estimates and
productivity projections. The government and military encourage contractors to use
metrics which implement the Software Engineering Institute (SEI) Capability Maturity
Model® - Integrated for Systems Engineering/Software Engineering (CMMI) (Reference
1) or equivalent methodologies during source selections and by making measurement a
contractual requirement.
Metrics play an important role in setting the target effort, cost, and schedule of a
project and in gauging the progress of a project toward these goals. Measurements are
related to cost estimation in three areas: project tracking, replanning, and calibration of
24

32. estimates, and estimating models. Recognition of these relationships is essential for
project success. In-process measures of effort, cost, schedule, and technical activities
collected during the project aid the project manager in accurately assessing the progress
of a project. Large deviations of the actual values from the planned values cause detailed
analysis of project performance, which frequently leads to actions taken early enough to
avoid subsequent inadvertent schedule and cost impacts. Actual measured values for
project attributes are used to reestimate project effort, cost, and schedule during the
project. This happens whenever project attributes change or unplanned events occur. For
example, new requirements are always added to the project. These reestimates provide
more accurate estimates for use in more efficiently managing the rest of the project.
Engineering practitioners are encouraged to define the set of information that they
need to execute the project and then gather only needed measurements. As the need for
more data elements arises, add only those measurements that fulfill the new need. One of
the worst detriments to any metrics program is to spend too much effort gathering too
many measures that require too much time and effort to analyze and are not even utilized.
Specific metrics used on a given project are chosen in full consideration of the overall
expense and impact of performing the measurement along with the value added to the
project or implementing organization.
The cost metric includes those costs associated with the economic performance of
the project. In addition to labor costs for various functional areas, the cost of a project
involves various items such as materials (including scrappage), tooling, subcontracted
processes and/or products and services, computers, terminals, workstations, Commercial
25

33. Off-the-Shelf (COTS) products and tools, support tools, and travel to customer sites. An
initial estimate is made during the proposal stage. Once the project starts, actual costs are
tracked by measurement collection time period. On software development projects, when
using the cost metric to calculate a cost of developing a source line of code (SLOC),
function point, component, subassembly, or object point, only the costs associated with
the development of the software (i.e., labor, development environment, tools,
workstations, etc.) are utilized. Costs associated with hardware and software products
procured for the end-user are not used. When using the cost metric to develop a
component, subassembly, or assembly, only the costs associated with the development of
the hardware (e.g., materials, subcontracted lower-level units, tooling, test equipment,
etc.) are useful. Inclusion of organic software is dependent, as all cost boundaries are, on
the cost Work Breakdown Structure (WBS) and subordinate documentation.
Table 2 in Appendix A shows sample measurements for CMMI Maturity Level 3.
(See Appendix A for Table 2.) For each process area, sample measures that can be used
to meet the measurement requirement for that process area are shown. These expanded
measurements are also applicable for projects or organizations adopting the CMMI
Continuous Representation and using staged equivalence to map to Maturity Level 3.
The core measurement set is the starting point for a project and can be tailored
while remaining consistent with other applicable policies. Tailoring is done to
accommodate specific customer requirements, the nature of the project (i.e., the activities
encompassed by the scope of the contract), the kind of product being developed,
26

34. applicable standards, or technology being applied to the project. All development
projects do some tailoring because the core measurement set contains options. So
choices have to be made. Also, additional specificity in definitions is sometimes required
as measurements are applied to real projects. For example:
• Software size may be measured in SLOCs, function points, feature points, object
points, or by some other defined size measurement.
• The software size count may be decomposed into categories of interest (i.e.,
newly developed, generated by a code generator, and reused).
• Test cases, procedures, or steps may be used to measure test size.
Each measurement collected, tracked, and analyzed as part of the project is
captured by phase. The metrics collected can be a single value or can be further refined
through classification and/or categorization. All metric tailoring decisions are
documented in the project measurement plan.
A reverse way of viewing tailoring is to address each measurement that will be
collected and define why it is collected, and what will be done with the information
developed from analysis of the data. If the measurement’s contribution to the execution
of the project cannot be shown, it is probably a candidate for tailoring.16
An Effective Metrics Process Model
Most organizations take measurements or metrics because they have the ability to
measure instead of determining why the information is needed. Unfortunately,
16
Software Engineering Institute (SEI), CMMI Product Team, Capability Maturity Model - Integrated for
Systems Engineering/Software Engineering, Version 1.1, [online article], CMU/SEI-2002-TR-004,
http://www.sei.cmu.edu/pub/documents/02.reports/pdf/02tr004.pdf, accessed 9 January 03.
27

35. measurement simply for the sake of collecting a number or statistic rarely makes a
process better, faster, or cheaper. A poor measurement can hinder a process if poor
decisions are based on the result of it. People at all levels of organizations continue to
collect measurements hoping that the metrics will shed light on the best way to provide a
product or service. Though fraught with good intentions, these poorly contrived
measurements add to the confusion of what should and should not be measured.
Most of the communications and information metrics of the Air Force Space
Command (AFSC) used to be taken because they had been collected for years, and were
thought to have a purpose. Previously, many metrics were not being used to make a
factual decision, but fulfilled a headquarters’ requirement to report on information by a
certain date every month. After a fairly extensive study, the AFSC Senior Communicator
(SC) changed the format and collection of these metrics while deleting the requirement
for many that had little or no value.
Like many discoveries, the process for metrics collection and analysis in this
directorate was the result of a leadership change. Communications metrics at AFSC
provided good information, since senior leaders did not complain about content or
format of the 30 metrics collected at headquarters. Haphazard metrics collection
continued until a number of new senior leaders asked why these metrics were being
collected and if they were the right ones for the organization. These questions caused
a complete review of the metrics collection, analysis, and reporting process. After
doing a thorough analysis of existing methods and an analysis of information on this
topic, it was decided that a common definition and set of criteria for good metrics
28

36. collection, reporting, and analysis would be used. Foremost in the quest for good
metrics was a definition of a “good metric.” Although a review of current research on
metrics found many definitions of this term, they could be summarized as one that
helps the right level of leadership make the right decisions in a timely manner, based
on fact rather than “gut feeling.”
Applying the Model - Establish and Validate Organizational Goals
With that definition in mind, the majority of authors note that the first step in
good metrics collection is understanding the goal. Rather than asking what should be
measured, ask what is important to the organization and the customer. Many
organizations struggle with this. But the Communications and Information
Directorate at AFSC did a thorough review of customer requirements and knew what
was important to the organization’s success. The SC directorate validated its
organizational goals and objectives with its customers, suppliers, and senior managers
when it published a strategic plan. Re-validated semiannually, this eight-page
document outlines the direction the unit is to take in the future. Notably missing from
the organization’s strategic plan was the connection of metrics to measure the
progress of these goals. In re-validating the strategic metrics plan, using metrics as a
tool to measure these goals, many noted that the goals were too general since they
could not be measured. These goals and objectives were revaluated thus ensuring that
each objective had a related measurement to guarantee progress.
Although goals are important to every organization, it can be hard to focus on
defining clear, measurable goals, based on what is important to customers. Upper
29

37. management is often doubtful about the value of spending resources defining such
goals. The Communications and Information Directorate at AFSC understood the
need for these goals but proceeded carefully by defining those goals that were most
easily quantified first.
Measures of system up-time rates and availability were clear targets with
measurable rates and long data histories. Once the goals provided useful decision
points, senior leaders were willing to define other goals of interest to the organization
and ultimately to the customer. Each organization must decide how many goals it
needs to effectively manage its resources and meet customer requirements. Through
trial and error, the organization found out that its customer requirements could be
summarized into ten measurable goals and forty more specific subgoals or objectives.
They provided a broad definition for what was important to the organization, while
the objectives provided actions to meet customer requirements. Each objective was
written to be clearly measurable. At least one associated metric was created for each
objective to provide decision-making data to senior management. Every organization
has a different approach to establishing goals based on customer requirements.
Regardless of the approach, it is vital that these goals are measured and quantified in
language that senior management can grasp and fully support.
Creating a Metrics Plan
The Communications and Information Directorate had a good data collection
program, but the analysis and use of this information was lacking. Although the aim
of these metrics was to measure an important area, the number of metrics continued
30

38. to grow, while the analysis was nonexistent. A plan was then devised to validate the
purpose of each metric. Rather than modify existing metrics, the metrics program
was overhauled. Most of the cost, schedule, and performance metrics were valid
because they directly measured the mission. However, the metrics process to gather
and analyze this information needed updating. An overall metrics philosophy was
defined as an adjunct to the strategic plan and noted that each new metric had to have
basic information associated with it, making it useful to the right people at the right
time. Although simple, this broad overview caused users in the organization to think
before creating or approving a metric. It also indicates the conditions under which
the metric can remain valuable. This makes the process better for semiannual review
of metrics, because the useful criteria are spelled out and unused metrics are deleted
or replaced.
Reviewing the Metrics Plan
In creating the metrics plan, it was noted that there may be other factors that
had not been considered when defining each metric. To review the data objectively,
the data was surveyed by collectors and senior leaders to see if they understood why
each metric was collected. The results were enlightening and helped create a valuable
metrics program. In this survey, questions about the metrics’ perceived usefulness,
its ability to help in decision making, goal of the metric, tolerances set for the highest
and lowest acceptable values, and timeliness of feedback based on analysis of the data
were asked. Users could have been interviewed instead of taking a survey but
anonymous answers are more honest. One survey for each metric to each of three
31

39. groups was distributed. One survey was given to data collectors and a different one
was sent to senior managers assigned to make decisions on each metric. The third set
of surveys was distributed to the metric owners that designed the metric and are
assigned to analyze the data. These surveys were used as the control group because
those who designed the metric have to understand why the metric is important in
decision making. Through this survey, raw data on specific problems and kudos
associated with each metric were obtained. Although specific problems were
addressed, the main reason for this analysis was to assess the overall usefulness of the
metrics program. This analysis was of greatest value to senior managers that make
final decisions on the kinds of metrics to be collected in the organization.
The first trend analysis indicated that one-third of the metrics were not useful
in decision making. Some had no reason at all for being collected. Other
measurements had outlived their usefulness. Also, few received timely feedback
from those who were assigned to analyze the data. All these indicators led to a
metrics program that provided much data but little useful information. Before
implementing the metrics plan, many believed that their metrics were the correct
measurement. Modifications were made to existing metrics to streamline and
standardize collection processes, and several metrics were deleted. After the new
metrics passed a trial period, senior managers were confident that the new metrics
provided information that was useful and necessary in making decisions. A plan is
proven only after it is implemented. Senior managers knew this. So after careful
32

40. planning, they began to provide policy and process clarification to those collecting
and analyzing data.
The right metrics later became obvious in the goal-definition stage. Originally,
goals were stated only in terms of up-time rate and easily measured quantities. These
were not the best metrics, but they were the easiest to collect. Many originally were not
used because they were not easily inserted in a bar or Gantt chart. It soon became known
that by defining the goal, the metric is obvious, instead of defining an easy metric and
trying to make a goal based on it. After much discussion, the goal became “reduce
operations and maintenance costs by 20 percent while maintaining equal or better service
to the customer.” With this clear, measurable goal in mind, metrics were formed that
measured total system cost, cost per capita, and cost per megabyte of data. Cost was
determined by dollars and manpower required. The purpose of this goal was obvious.
The decisions associated with these metrics were no longer fuzzy. These costs are
comparable to in-house and contract labor costs. The organization concluded that the
most useful metrics were those that compared two or more quantities rather than simply
reporting finite measurements. When these metrics were compared with the up-time
rates, some very significant savings opportunities were discovered. A number of
important changes were made based on the new metrics. Therefore, in looking at
network status and up-time rates on servers, it was decided that a 100 percent up-time
rate was not cost effective, based on the analysis of the cost of up-time vs. network
availability and efficiency. Also by comparing costs per capita with costs per megabyte,
many changes were made to consolidate information processing operations, thus saving
33

41. maintenance man-hours and server costs. By following a systematic process, the
organization could define clear, measurable goals and obtain data vital to the decision-
making process.
Throughout the available literature, researchers note the complexity of creating
metrics that are easily understood yet help the right management level make timely
conclusions based on fact. The difficulty is that the question of how to measure a process
is continually asked as opposed to determining what decisions to make. If organizational
goals are written clearly and are measurable, creating a metrics program becomes easy.
A successful metrics program ensures that data is collected and analyzed consistently,
and most importantly, that the right personnel are making good organizational and
corporate decisions based on fact.17
The Architecture Tradeoff Analysis Method (ATAM)
The ATAM is a useful econometric technique for analyzing and evaluating
software architectures. The SEI developed and improved this method over the past
several years. It not only is used to evaluate architectural decisions against specific
quality attributes, but it also allows engineering tradeoffs to be made between possibly
conflicting system quality goals. In this fashion, the ATAM evaluation detects areas of
probable risk in meeting quality goals within the architecture of complex software-
intensive systems.
17
Capt. Thomas Augustine, Dr. Charles Schroeder, An Effective Metrics Process Model, STSC CrossTalk,
June 1999 Issue, [article on-line], http://www.stsc.hill.af.mil/crosstalk/1999/06/index.html, accessed on 16
December 2002; Internet.
34

42. There are two key factors in any source selection that may prevent the ATAM
from being applied. First, to maintain fairness, the communications between the acquirer
and all offerors must be formal and consistent. Second, the solicitation process is not
conducive to iteration.
An ATAM-based evaluation may be used to reduce risk in a system acquisition.
Used appropriately, that this technique can help the acquirer select a supplier that is more
capable than other suppliers of developing a software-intensive system that meets its
quality goals.18
The Cost Benefit Analysis Method (CBAM)
The creation and maintenance of a complex software-intensive system involves
making a series of architecture design decisions. The Architecture Tradeoff Analysis
Method (ATAM) gives software developers a framework for understanding the technical
tradeoffs they face as they make design or maintenance decisions. But the biggest
tradeoffs in large and complex systems usually deal with economics. The ATAM does
not provide guidance for analyzing economic tradeoffs. Organizations need to learn how
to invest resources to maximize the gains and minimize the risks. When others have
addressed economic tradeoff issues in the past, the focus has always been on the cost of
building the system, not the long-term costs of maintenance and upgrade. But the
benefits that an architecture decision brings to an organization are more important than
the costs.
18
John K. Bergey, Matthew J. Fisher, Lawrence G. Jones, Use of the Architecture Tradeoff Analysis
Method (ATAM) in Source Selection of Software-Intensive Systems, Technical Note, June 2002, CMU/SEI-
2002-TN-010, [article on-line], http://www.sei.cmu.edu/pub/documents/02.reports /pdf/02tn010.pdf,
accessed 7 July 2002; Internet.
35

43. Because the resources for building and maintaining a system are finite, there is a
rational process for selecting between architectural options during the initial design phase
and during subsequent periods of project upgrade. These options have various costs,
consume different amounts of resources, implement different metrics each of which give
some benefit to the organization, and have some built-in risk or uncertainty. To examine
the effects of these options, economic software models are needed to take into account
costs, benefits, risks, and schedule implications.
The first version of the Cost Benefit Analysis Method (CBAM) was a quantitative
economic technique for making design decisions. This method shows that architectural
strategies (ASs) influence the system quality attributes (QAs) and in turn provide the
stakeholders of the system with economic benefit. The benefit added to the system from
AS implementation is called utility. Each AS allows a specific level of utility to the
stakeholders. Also each AS has cost and schedule implications. This information can
help the stakeholders in selecting ASs based on the return on investment (ROI), which is
calculated as the ratio of benefit to cost.
While this method is reasonable in theory, there are several problems with its
practical implementation. One problem is that QAs such as performance, modifiability,
and usability are abstract entities to the stakeholders. These make it hard for stakeholders
to interpret QAs consistently. Stakeholders also have trouble quantifying the expected
AS utility level. Thus highly variable judgments can result if the interpretations cannot
be calibrated with the current system’s business goals. These problems are notable as the
36

44. CBAM is prototyped in a real-world setting. In response, a second version of the CBAM
has been created.
The software architect/decision maker has to maximize the benefit derived from
the system and minimize the cost of implementing the design. The CBAM begins where
the ATAM ends. It still depends on the artifacts produced by the ATAM. Figure 2
below depicts the context or environment for the CBAM.
Figure 2 – Context of the CBAM
Because the ASs have technical and economic implications, the software system
business goals influence the ASs used by software architects and designers. The
technical implications are the software system characteristics (generally QAs). The direct
economic influence is the implementation cost of the system. However, the QAs also
have economic ramifications because of the benefits that are derived from the system.
The ATAM includes a set of key architectural decisions relevant to the QA
scenarios that come from the stakeholders. These decisions result in unique QA
responses, such as levels of availability, performance, security, usability, and
modifiability. But each of those architectural areas also has related costs. For example,
37

45. using redundant hardware to get a desired availability level has a cost, and checkpointing
to a disk file has a separate cost. Furthermore, both of these architectural decision areas
result in different measurable levels of availability with some real value to the
development organization. Perhaps organizations believe that stakeholders will pay more
for a highly reliable system (for example, software that controls anti-lock brakes in an
automobile or medical monitoring software). Or maybe it will get sued if the system is
not always available (for example, a telephone switch).
The ATAM displays the architectural decisions made in the system and connects
them to business goals and QA response metrics. The CBAM builds on this foundation
by discovering the costs and benefits of these decisions. The stakeholders can decide
whether to use redundant hardware, checkpointing, or some other strategy to realize the
system’s optimal availability with this information. Or they can choose to invest their
finite resources in some other QA. Maybe higher performance will have an increased
benefit to cost ratio. Most systems have a limited budget for new development,
maintenance, or upgrade so each architectural choice is competing with all others for
inclusion.
The CBAM does not make decisions for the stakeholders. The CBAM simply
helps with the derivation and documentation of the costs, benefits, and uncertainty of a
group of architectural investments. Also it gives the decision-makers a foundation for
using a rational decision-making process that fits their needs and risk disinclination. The
idea of the CBAM is that ASs affect the system QAs and in turn provide the stakeholders
of the system with a level of utility. But each AS also has associated costs and takes up
38

46. implementation time. With this information, the CBAM lets the stakeholders choose ASs
based on the ROI they provide.
Incorporating Uncertainty
Expanding on the CBAM process can create a more cultivated and realistic
CBAM version. Secondary data can be added about risk assessment, uncertainty, and
development resource allocation. Each category of secondary information possibly
influences the investment decisions being considered. Therefore, the ways the method is
augmented must be considered carefully for correctness and for practicality.
The CBAM depends on stakeholder judgments for valuations of software benefits
and costs. But these analyses will obviously be uncertain, due to variations in beliefs and
experiences. One way to think intensely about the collected results unpredictability is to
gather and consider the risks implicit in the estimates that have been made. To do this,
some kind of risk appraisal exercise must be performed. These risks fall into four
categories:
1. risks influencing the cost estimate of a strategy being assessed
2. risks affecting a stimulus-response characteristic or a strategic utility estimate in
the context of a scenario
3. risks that affect stimulus-response attributes of other scenarios or QAs not
previously considered. (These risks involve the side effects rather than the
intended AS effects.)
4. risks linked with project management and schedule.
Given this risk information, it cannot be assumed that the AS costs, the resulting
QA response levels, and the related utility levels are all known and derived from the
stakeholders with certainty. Therefore, the indefiniteness of those three dimensions can
39

47. be dealt with clearly. The final result of this process is to associate with each cost,
response, and utility a range or distribution of values, rather than one point value for
each.
The stakeholders must assess the probability and effect of each risk and use this
to calculate a delta expected cost, delta expected response, and utility value for each risk
influencing each AS. When this technique is implemented, the result is a range of
values, not a single point. Probability calculations are then used to find the chance of
the ROI ranking changing.
A process flow diagram for the CBAM is now developed as depicted in Figure 3
on the next page. The first four steps are annotated with the relative numbers of
scenarios that are considered. The number of scenarios analyzed steadily decreases. This
is the CBAM’s method of centering the stakeholders’ valuable time and attention on the
situations that are of the greatest potential in terms of ROI.
Any technique for improving the quality of a system or making strategic decisions
about the system has a cost. This is the cost of using the method. There are also some
benefits such as better, more informed decisions, a formal process that the stakeholders
follow, and greater stakeholder buyin. Usually the stakeholders like to maximize the ROI
of running the CBAM just as much as they like to maximize the system ROI itself. So it
is vital that time spent on the CBAM is kept to a minimum and is focused on the most
important scenarios and ASs.
40

48. Figure 3 - Process Flow Diagram for the CBAM
While the ATAM uses individual scenarios, the CBAM uses a series of scenarios
that are generated by changing the response values. This elicits the utility-response curve
concept. Every stimulus-response value pair in a scenario provides utility to the
stakeholders. The utility of the various possible values for the response can be compared.
For example, the stakeholders may only value a real high availability in response to a
41

49. failure scenario a little more than a moderate availability level. But a low latency
response to a particular performance scenario is valued much more than a moderate
latency. Every relationship between a set of utility measures and a corresponding set of
response measures is portrayed as a graph, i.e., a utility-response curve. Some examples
of utility-response curves are shown in Figure 4. In each example, the point labeled “a”
represents one response value, and the point labeled “b” represents another value. The
utility-response curve shows the utility as a response value function.
Figure 4 - Some Sample Utility-Response Curves
The utility-response curve shows how the utility derived from a certain response
changes as the response varies. As shown in Figure 4, the utility can vary non-linearly,
42

50. linearly, or even as a step function. For example, graph (c) displays a steep rise in utility
over a narrow variance in a QA response level. An example might be better depicted by
graph (a), where a modest change in the response level results in only a very small
difference in utility to the user.
Eliciting the utility properties from the stakeholders is a long and difficult
process. To make the process practical, the stakeholders are asked just to provide round
approximations of these curves by using five values of the QA response for the scenario.
To form the utility response curve, the QA levels for the best-case and worst-case
situations are calculated. The best-case QA level is that level above which the
stakeholders foresee no further utility. For example, if the system responds to a user in
0.1 second, that is seen as instantaneous. Improving it more to make it respond in 0.03
seconds gives no utility to the user. Similarly, the worst-case QA level is a minimum
value above which the system must perform. Otherwise, the system is not useful to the
stakeholders. The best-case and worst-case levels are attributed utility values of 100 and
0, respectively.
Then the current and desired utility levels for the scenario are calculated. The
respective utility values (between 0 and 100) for the current and desired cases are derived
from the stakeholders, using the best-case and worst-case values as reference points (e.g.,
the utility values are currently half as good as needed, but if the desired QA level is
reached, then users would have 90% maximum utility; thus, the current utility level is set
to 50 and the desired utility level is set to 90). The curves are generated for all the
scenarios in this way.
43

51. Inter-Scenario Significance
Different situations within a given system have different importance levels to the
stakeholders and therefore separate utilities. To depict the relative significance of each
scenario, a weight is assigned through a two-step voting exercise. In the first step, the
stakeholders vote on the scenarios to order them. This voting depends on the expected
response value for each scenario. Then the stakeholders give a weight of 1 to the highest-
rated scenario and a fractional value to the other scenarios contingent on their
comparative importance. If, at a later date, additional scenarios need to be added, they
can be assigned a weight indicated by their relative importance. The stakeholders, via
consensus, guarantee that the scenario weights agree with their intuition.
Architectural Strategies
The architect’s task is to determine the Architectural Strategies (ASs) necessary
for moving from the current QA response level to the desired or even best case level. A
portion of the CBAM is dedicated to the enumeration of a group of such ASs. For each
strategy, the following information can be elicited:
• the expected value of the response in the scenario. (The utility of the expected
value is calculated using interpolation from the four values already received from
the stakeholders.)
• the influence of the AS on other characteristics of interest
• a cost estimate for implementing the AS (an ROI can also be calculated).
Side Effects
Each AS impacts not only the QA from the scenario being considered but also
other QAs (which is what produces architectural tradeoffs!). It is important to decide the
utility of the additional side effect attribute responses that arise when the AS is applied.
44

52. In the worst case, a new scenario is created and its utility-response curve is created by
deriving the best, worst, current, and desired response values from the stakeholders and
interpreting the expected utility. In reality, however, if the QA is important to the
stakeholders, it occurs in one of the other scenarios, and the utility-response curve for that
response is already constructed. In that case, the only thing left for calculating would be
the expected utility of that QA for the given AS. Notice that the expected utility of a
certain characteristic may be negative if the AS is designed to emphasize an attribute in
conflict with it. Once this additional information has been discovered, the benefit of
applying an AS can be calculated by adding its applied benefits to all relevant QAs.
Determining Benefit and Normalization
The overall AS benefit can be calculated across scenarios from the utility-
response curves by totaling each scenario benefit (weighted by the importance of the
scenario). For each AS (i), calculate a benefit, Bi as follows:
where bi,j is the benefit attributed to AS (i) from its effect on scenario j, and Wj is the
weight of scenario j. Referring to Figure 4, each bi,j is calculated as the difference in
utility that is caused by the AS in this scenario. This “delta utility” is calculated as
follows:
bi,j = Uexpected - Ucurrent, the utility of the AS expected value minus the current system
utility compared to this scenario. The effect of multiplying the weight Wj is to normalize
the utility value by the relative importance of the various scenarios.
45

53. Cost
To use the CBAM in making economic decisions, the benefit information has to
be combined with cost estimates. For each AS considered, an implementation cost of that
strategy is calculated using a cost model that is compatible with the system and
environment being developed. The CBAM is independent of any particular cost model.
No cost model is prescribed.
Calculating ROI
The ROI value for each AS is the total benefit ratio, Bi, to the cost, Ci, of
implementing the strategy, as shown in the equation below.
Using this ROI score, the ASs can be rank ordered. This rank ordering can be
used to find a desirable order for implementing the various strategies. Also note that it
may be important to take the shape of the curve into account when ranking ASs.
Consider curves (a) and (b) in Figure 4. Curve (a) levels off as the QA response
improves. In this case, it is probable that a point is reached past which ROI decreases as
the QA response improves. Now spending more money will not yield a measurable rise
in utility. Or consider curve (b) for which a small improvement in QA response yields a
very significant increase in utility. In this case the ROI increases with improvements in
the QA response. Therefore an AS whose ROI is too low to consider may rank much
higher with a small improvement in its QA response.
46

54. Dealing with Uncertainty
Within the CBAM, there are some uncertain variables that are stakeholder
judgment estimates:
1. Inter-scenario weights - This variable defines the one scenario’s relative
importance with respect to all others. The relative importance is quantified either
as votes or as absolute ratios (e.g., the lowest priority scenario has a vote of 1 and
the other scenarios have values more than 1, indicating their relative values).
These weights are dimensionless numbers greater than 0.
2. Intra-scenario utility - This is depicted by a utility function, for a particular
scenario, for changing levels of the stimulus/response characteristic. (Currently
five levels are used: worst, current, expected, desired, and best.) The unit is
“utils” and defined between 0 and 100.
3. QA level realized through implementation of an AS - This variable defines the
estimated response characteristic that can be achieved by implementing an AS.
This number’s dimension depends on the attribute being measured.
4. Cost of implementing an AS - This variable defines the implementation cost of
an AS, measured in person-months or dollars.
Given the fact that the values elicited for these variables are certain to be
uncertain, an ROI score will eventually be attained, R(i), that is uncertain. The score (for
each AS) can be represented by a probability distribution that is either uniform,
triangular, or normal. R(i) is the mean/mode value, R(i,max) and R(i,min) will be the
bounds of the uniform/triangular distribution, and σ(i) is the standard deviation for a
normal distribution. Along with the rank ordering, the certainty or uncertainty regarding
this rank ordering can be quantified. The probability that R(i) is greater than R(j) can be
calculated according to:
47

55. where fi(x) is the probability density function of R(i) and19
Quantitative Economic Models to Qualitative Engineering Judgments
This research attempts to apply financial portfolio analysis techniques to the task
of selecting application-appropriate technologies from those available in the marketplace.
The problem structures are similar in that the intuitive guidance is encouraging. But the
analysis techniques of portfolio analysis assume precise quantitative data of a sort that
cannot be realistically obtained for security applications. This is a common challenge in
applying quantitative economic models to software engineering problems, and the
following considers methods or techniques to address the mismatch.
At the First International Workshop on Economics-Driven Software Engineering
Research (EDSER) it was proposed that using portfolio analysis assists software
engineers in selecting security technologies. Security technology selection constraints
were briefly described which suggested that portfolio analysis can be used to select a
reasonable suite of technologies for a system. In ongoing work that possibility has been
investigated. A mismatch between the kinds of data that can reasonably be expected for
security designers to create and the kinds of data portfolio analysis tools are designed to
operate with has been discovered. It appears that this kind of mismatch happens often in
attempts to apply economic models to software engineering, so this research resolves this
19
Rick Kazman, Jai Asundi, Mark Klein, Making Architecture Design Decisions: An Economic Approach,
Technical Report, September 2002, CMU/SEI-2002-TR-035, [article on-line],
http://www.sei.cmu.edu/pub/documents/02.reports/pdf/02tr035.pdf, accessed on 15 November 2002.
48

56. mismatch directly. The security technology selection problem identifies some of the
issues in using portfolio analysis. This research suggests three approaches to resolving
these limitations and suggests techniques so that these examples serve as models for the
EDSER community.
Portfolio Analysis and the Security Technology Selection Problem
The difficulty faced by the system security engineer is to protect system resources
from attack, especially to decrease the loss risk. Threats send attacks that find system
vulnerabilities to gain or deny access to system critical files. A system security engineer
must choose security technologies that minimize any expected risk. The set of selected
security technologies is the security portfolio.
The system security risk depends on the relevant set of potential attacks and the
results of a successful attack. The important threats (and thus the attacks they may
launch) and resources are variable among systems, so security portfolios also vary. Cost
constraints and performance needs keep software engineers from selecting all the best
technologies. Cost constraints consist of both short-term costs, like the initial purchase
and installation costs, and long-term maintenance costs. Maintenance is often the largest
cost.
Other constraints also influence the selection of a security portfolio. For example,
security engineers often want some amount of coverage against various attacks. There
ought to be at least one countermeasure or metric for every identified threat. However, in
reality some threats are not covered by countermeasures. They may not occur, or the
outcome of an attack can be less than the security technology cost. Because most
security technologies are not 100% effective, engineers require extra redundant
49

57. countermeasures against an attack. For example, intrusion detection systems allow more
security against attacks that penetrate system firewalls, and encrypted data storage adds
security against undetected intruders.
Security Technology Selection as Portfolio Analysis
There are many commonalities between building a financial portfolio and
selecting security technologies for an information system. The objective of both is to get
the set of investments that best supports investment objectives. For financial portfolio
investment, the objective is the desired rate of return. For security systems, the objective
is an adequate risk limit. Security technologies can be viewed as investments and the
degree to which the technology prevents damage from attacks is the return on investment.
The idea of financial portfolios is that the desired ROI variance is less than individual
portfolio elements, therefore the risk of getting that return is lower. Similarly, a set of
security technologies reduces the system risk, and the constraints help in modeling the
expected rates of return. The technology combination decreases the consequence of a
successful attack.
Although portfolio analysis is very useful in modeling the security selection
problem, the technique has its limitations. Financial portfolios contain financial
instruments with various ROIs. These values lend themselves to quantification. For
security portfolios, however, the acceptable-risk objective demands significant subjective
evaluation.
The portfolio analysis analogy helps to establish the security technology problem
and informally suggests heuristic approaches to a solution. However, more of the
knowledge available in the financial portfolio needs to be used. The following gives
50

58. some concepts about coping with the gap between security information precision and the
mathematical expectations of portfolio analysis tools.
Mismatch Between Model and Available Data
The biggest hindrance to using portfolio analysis is the lack of quantifiable data.
This issue appears as two parameters of the security model, threat expectation and
security technology effectiveness. Software engineers pick technologies based on threat
expectations or their anticipation of attacks. But there is rarely statistical information to
back up these expectations. Threat expectations are usually phrased as “We are more
likely to get hit by a disgruntled employee than by an internet hacker” or “I really worry
that someone will …..”. Statements like this reflect attack expectations, but they do not
translate into quantifiable probabilities.
Security technology effectiveness is also very hard to quantify. Technology
payoff depends on the attack type, configuration and maintenance complexity, design,
system characteristics, etc. Security professionals comment on the relative degree of
effectiveness. For example, statements, such as “Technology ‘x’ is more effective than
technology ‘y’”, or “Technology ‘x’ is very effective against such and such type of
attack”, are typical. Ultimately, the security engineer decides based on partially ordered
qualitative information.
Addressing the Mismatch
A significant portion of the mismatch between security technology data and
financial analysis techniques comes from the fact that security technology data is
displayed on ordinal scales (“X is more effective than Y”). But the analysis methods are
51

59. tailored for data shown on a ratio scale. Consider three basic approaches to solving this
discrepancy between the data and the analysis tools:
• increase the data’s precision enough to convert the qualitative rankings to
quantitative metrics
• arrive at analysis techniques that require less precision
• demonstrate that analysis techniques of interest maintain the relations of the
qualitative data
Converting Qualitative Information to Quantitative Ranges
Perhaps the simplest way to encourage the security professionals to supply a bit
more information to establish estimated quantitative ranges. Security professionals may
be able to derive upper and lower bounds if they cannot provide precise threat
probabilities and effectiveness measures. Each threat could be characterized by the
bounds on its occurrence probability. A security professional may be willing to say
something like “I think there is a better than 50% chance that someone will try to modify
our web pages, but maybe not more than an 80% chance” when they cannot say “the
probability of this attack is 67.3%.” Reasonable technology effectiveness commitments
can be in the form of “I believe this security technology will stop between 40 to 60
percent of the IP spoofing attacks.” When the quantitative limits have been set, the
portfolio analysis techniques are used for best-case and worst-case analyses to handle
sensitivity analysis and partial ordering validation.
The drawback of using upper and lower quantitative bounds is that the delta
between the upper and lower bounds can be too high to get useful results. Large ranges
weaken the results. A lower bound of 10% and an upper bound of 90% does not provide
52

60. enough useful information. Also, the range estimates are not independent of each other.
Another limitation is that humans are famous for making bad probability estimates so the
upper and lower bounds are very unreliable. Eventually, security researchers will
establish threat probabilities based on empirical evidence, but estimating the security
technology effectiveness will still be left undone. Other metric techniques are required
before there is stakeholder confidence in the results.
Analysis Based on Partial Orderings
Another method for using qualitative data is decision analysis techniques that use
partial orderings. Our problem of choosing security technologies is similar to the two-
sided matching problem. An example of this is the national intern selection problem.
Each year new medical school interns submit a partially ordered hospital list where they
prefer to work. Hospitals also create a partially ordered interns list of which they would
like to come work at those hospitals. The issue is finding a stable matching of interns and
hospitals. Two-sided matching problems are widely analyzed in game theory and
economics.
The two-sided matching problem in security is between security technologies and
threats. A stable matching so that each threat is paired with a security technology needs
to be found and categories of technologies are selected. The security portfolio might not
set up a one-to-one match between technologies and threats. This could require an
extension of the basic two-sided solution.
Establish Validity of Using Quantitative Techniques after Scale Transformations
A third technique is to try the idea of transform spaces. A common technique in
mathematics and engineering is converting data from normal space to a transform space.
53

61. This carries out computations in the transform space and converts the results back to
normal space. This is typically done because the computation in transform space is so
much easier than the corresponding normal-space computation that it offsets the cost of
the transformations. The most common example is logarithmic space: adding logarithms
corresponds to normal-space addition. When utilizing a slide rule does this, the
transformation cost is low. The LaPlace and Fourier transforms are other familiar
transforms used in this manner for specialized applications.
These common situations involve transformations on values rather than on a
measurement scale. But they raise the interesting question of whether these techniques
can perform a scale transform on ordinal data to another scale, use analysis tools
designed for that scale, and extract a useable outcome. In a more-simplified form of the
model (ignoring resource constraints and the differential technology effectiveness against
various threats, among other things):
• Let T be a threat levels vector, expressed on an ordinal scale.
• Let E be a security technology effectiveness vector, expressed by an ordinal scale.
• We seek a portfolio selection function So(T,E) that operates with ordinal scales
and returns a set of indices {ij}corresponding to the technologies to include in the
portfolio
There is no function So of this kind, but there is a function Sr that performs a
similar function with ratio scales. Are there mappings MT and ME with the properties
MT(T) and ME(E) expressed in ratio scales such that Sr(MT(T),ME(E)) yields results
equivalent to So(T,E) after applying an inverse mapping on the result? This is very
possible for functions Sr that calculate only monotonic transformations and max/min
54