Closure of Rocky Flats, a former nuclear weapons production plant involves decommissioning of wireline-based telecommunications, networking and applications infrastructure from approximately 500 buildings, replacing it with wireless networking and Voice Over IP systems for use by a workforce removing the physical structures and their nuclear materials.

1. Creating the Foundation for IT
Project Portfolio Management
at Rocky Flats Environmental
Technology Site
CH2M HILL’s Communication Group (www.ch2m.com) provides information and
network services to the Rocky Flats Environmental Technology Site (RFETS)
(www.rfets.gov) in support of the site’s closure activities. Closure of this former
nuclear weapons production plant involves the decommissioning of the wireline
based telecommunications, networking and applications infrastructure from
approximately 500 buildings, replacing it with wireless networking and Voice over IP
systems for use by the workforce removing the physical of the structures and their
nuclear materials. Several 100 database, web, and COTS applications must be
maintained, enhanced, and decommissioned during the closure process. Post–closure
many of these applications will be used for long–term stewardship of the site.
The delivery of this diverse set of projects takes place through the Program
Management Office. Project managers and their staff’s provide support, guidance,
and project coordination to functional managers and their staffs totaling ~120
professionals and technicians. Project planning, execution, control, and reporting are
performed through portfolio management, Balanced Scorecard, Earned Value
Analysis, and project selection and measurement processes. This paper focuses on
the management of software development projects and the issues associated with
making “level of effort” development tasks visible to the management team.
What is Project Portfolio Management?
Project portfolio management provides a view of projects to reveal redundancies,
allocate resources, and report progress to plan across all the projects, not just a few
[16]. More importantly it provides an “investment management” view of the
projects. Not just how much a project will cost but also the anticipated risk and
return on investment compared to other projects in the portfolio.
Before this information can be used in decision–making, the portfolio of projects
must meet several criteria. Performance data must represent the actual value
delivered from the investment. The performance metrics must be connected to the
strategies of the organization. The project selection process must be guided by the
organization’s strategic goals not just the tactical needs of individual groups.
The processes described in Figure 2 are the foundation of portfolio management.
Without strategic goals, corresponding metrics, Project Portfolio Management
provides little more than fancy graphs, charts, and numbers.
Legacy Project Management Environment
Prior to CH2M HILL’s assumption of the delivering of the IT functions a variety of
forces driving the project management activities were in place. These forces, shown
in Figure 1, were the motivation for the introduction of Project Portfolio Management
(PPM) and its supporting processes. [15]
1

2. Driving Force Consequence
High demand for
immediate delivery of
applications,
telecommunications, and
infrastructure services
arose daily.
Documentation of this
demand was located in
several systems, none of
which were interoperable.
Overall demand could not
be coordinated by
management from a
centralized location.
Localized planning
activities were developed,
further isolating this
information.
Staffing levels were being
reduced as part of the
closure plan. Coordinating
work demand with
resource availability was
done on an ad hoc basis,
within a small planning
window.
No “re purposing” of staff
could be done, since an
integrated resource
utilization management
plan could not be
developed for the
workload.
A broad mix of clients,
funding sources,
priorities, and decision
makers created a poor
delineation of the
relationship between
suppliers and consumers
of IT services.
Correlating needs across
technologies, user
communities, and delivery
systems could not be
done.
Needs that might benefit
a broader community
could not be identified.
Project selection process
was ad hoc. No alignment
between project priorities
and the IT strategy for
providing these services
and delivering value.
Project selection based on
local departmental needs
rather than a broader
business needs.
No feedback on the
business benefits
available to refine the
selection process.
No centralized project
inventory or repository or
tracking system to enable
the enterprise valuation
of projects.
Individuals held projects
in “private” locations
making open and visible
discussion of priorities
and resource integration
difficult.
Figure 1 – Forces driving legacy management of project portfolios
These forces create a “gap” between the planning of IT projects and the execution of
these project plans. The planning of a portfolio of projects, although not
straightforward, is only half of the solution. Execution of these plans requires the
definition of priorities, access to resource requirements across the collection of
projects, and an understanding of the consequences of any resource or priority
2

3. decision for the portfolio, not just individual projects. In the absence of this process
the most vocal user usually receives the highest priority, at least until a crisis is
encountered and the importance of a specific user is altered from of the current
priorities and resource availability.
Stephan Covey calls this “Living in Quadrant I,” where everything is urgent and
important. [7] This is the quadrant of crises, pressing problems, deadline driven
projects, intervention meetings, and few preparations. To gain control of a portfolio
of projects, the project management team has to “Live in Quadrant II.” This is the
quadrant of preparation, prevention, value and priority clarification, planning, and
group empowerment. To live in Quadrant II we have introduced four (4) systems
described in Figure 2.
These systems are necessary, but they alone are not sufficient to gain control of the
IT project portfolio and deliver value. [6] A vision of an effective IT department is
needed. One in which resources can be allocated with ease. One in which work is
planned on a daily, weekly, monthly, and quarterly basis, with the concurrence of the
stakeholders and the suppliers of services and products. One in which a reliable
projection of the labor and material costs is provided by the system with the
concurrence of all participants. One in which risk assessment and intervention is an
everyday activity.
The Foundation of PPM in the IT Services Domain
Successful management of IT project portfolios assumes that the best candidates for
success are selected for deployment. To identify candidate projects, the strategic
goals of the organizations must be identified. From these goals, initiatives are
created. Collections of projects supporting the initiatives are then assembled. Metrics
for the projects are installed, followed by data capture and assessment. Feedback
from the assessment is then used to verify the impact of the strategy on the
organization.
A portfolio management approach used to address the gaps shown in Figure 1 moves
the management process away from the Quadrant I’s external drivers and toward
Quadrant II’s process management and alignment. This control–based approach
make use of four (4) management techniques:
A Balanced Scorecard strategy to define priorities and establish a
connection between every project and a specific strategy and
objective.
A public registry of projects, resources, and deliverables housed in
Microsoft Project 2002 Server.
An Earned Value performance reporting and measurement processes
to make visible performance metrics for cost and schedule.
Processes to select, classify, measure, and guide the implementation
for collections of information technology projects.
Figure 2– Creating a foundation for Project Portfolio Management
Balanced Scorecard
In order to “manage” a portfolio of projects, these projects must be connected to the
strategic goals of the site and its closure activities. Balanced Scorecard (BSC) is a
3

4. management system that clarifies vision and strategy and translates them into
actions. These actions are defined through four perspectives: Financial, Customer,
Internal Business Processes, and Learning and Growth. [2] A “strategy map” is
constructed which states specific objectives for the deliverables from each
perspective. Such a map is shown in Figure 3.
Although BSC may appear to be an esoteric management tool it is a powerful
starting point for Project Portfolio Management. [10, 12] BSC answers the questions:
“Why are these projects being considered?” “How does this project support a
strategic direction?” “What is the value of this project to the organization beyond
monetary returns?”
In some organizations there is a distinction between “public” projects and “private”
projects. At CH2M HILL in support of RFETS, we have only public projects. If a
project cannot be aligned with a BSC objective, it cannot continue to be funded. [5]
The Balanced Scorecard consists of four “perspectives:” [10]
 Business results – what are the objectives of our efforts? How will we know when
we are successful?
 Customer expectations – what words do our customers use when they are ask for
our products and services?
 Internal processes – what work processes must be in place to deliver value to the
customer?
 Strategic enablers – what behaviors, attributes, facilities, and resources must be
in place before internal processes can function, customer needs can be heard,
and value delivered to the customer?
4
StrategicEnablers
L
Internal
Processes
BusinessResults
SiteExpectations
CompetencyCompetencyCompetencyCompetency
……to reduce overallto reduce overall
costcost
SustainSustain
requiredrequired
servicesservices
MM
cc
Build a highBuild a high
performance, closureperformance, closure
oriented IT cultureoriented IT culture
““Do the rigDo the rig
Operating ExcellenceOperating ExcellenceOperating ExcellenceOperating Excellence
Reduce cReduce co
Improve IT proceImprove IT proces
ManageManage
projectsprojects
effectivelyeffectively

5. Figure 3 – A Sample Strategy Map [1]
One aspect of BSC is difficult to accept at times — that every project in the portfolio
must support an objective in the strategy map. If a project does not support an
objective, then the question “why are we doing this,” must to be answered. If there
is not a strategy–based answer to that question, then the project is a candidate for
being dropped.
The organization needs the strength to assess projects in this manner. If this can be
done, all work performed by an IT function will support identifiable strategic
objectives.
Centralized Project Repository
The assembly of all projects in central location is a critical success factor for portfolio
management. By “all” it means ALL projects, no matter how small or obscure. [2]
We
adopted Microsoft Project Server to house the portfolio of projects. Although it may
not be the ideal choice, it provides a central repository for projects their tasks,
resources, and cost. It provides Earned Value Analysis of projects once they have
been baselined. Labor hours are captured through the Project Server as well as a
Defense Contract Audit Agency audited Time and Labor reporting system. These
hours and non–labor costs are applied to the project baseline to compute cost and
schedule variances and project performance indices.
The effort necessary to capture all activities into a central system should not be
underestimated. Organizational and human roadblocks appear that say, “… but what
you’re trying to measure is not a project.” The solution is to consider all activities to
be a “project,” even the level of effort tasks. This way all work is captured in the
Project Server. For “level of effort” activities (LOE), a project is defined in which LOE
tasks are recorded. No matter what the activity it can be found in the Project Server,
with the associated costs, resources, durations, and deliverables.
Earned Value Analysis
Traditional methods of measuring progress to plan produce poor results. We applied
Earned Value (EV) to projects and collections of projects. EV is used to measure and
communicate the physical progress of a project based on “work complete,” the effort
used to complete the work, and the non–labor costs incurred to complete the work.
EV is used to evaluate and control project risk by measuring project progress in
monetary terms for the actual value delivered to the customer.
1
The Strategy Map is a 3rd
generation Balanced Scorecard concept. It assembles in one place the
objectives, and their relationships in pursuit of the organizations delivery of value. Each member of the
organization should be able to “tell the story” of the role of each objective and how the projects support
the fulfillment of these objectives.
2
Small projects are captured as “to do” lists, rather than Gantt based projects. These “to do” lists have
resources assigned, deliverables, and due date. They act just like real projects, but with much lower
overhead.
5

6. Cost Variance
(CV)
Dollars (Labor Days) of
Spending
Behind or Ahead
of Schedule
Schedule
Variance
(SV)
(Days Behind or
Ahead of Schedule)
Schedule Variance
(SV)
Dollars
Planned
Value
(BCWS)
Budgeted Cost of
Work Performed
Earned Value
(BCWP)
Time
Cost
Actual Cost of
Work
Performed
(ACWP)
Figure 4– Simple Earned Value Analysis metrics
EV provides “leading” performance indicators that allow the project managers to
identify and control project problems before they become insurmountable. To do this
EV adds a third measure – the actual work accomplished. Measuring the actual work
accomplished provides greater insight into potential project risks. It also provides a
more accurate estimate of the completion schedule and cost estimates. With these
“leading” indicators, project managers proactively manage projects in ways not
available using only cost and schedule measures.
The key to deploying Earned Value for software development portfolio management
is to define a measure of “value” that indicates real progress not just the passage of
time and the achievement of major milestones. Technical performance measurement
and “testable requirements” is the method used to measure this value.
Technical Performance Measurement
The typical methods of measuring value are based on a binary event or some
subjective assessment of the progress. Both approaches fail to delivery accurate
measures of progress for software projects. This approach asks the question – “How
do we know the software will behave as specified?” If it does behave as specified,
then the development step is complete. If not, then rework is needed. [3]
We use “Technical Performance Measurement” to answer the question “How do we
know what ‘done’ means?”. [8, 13] Technical Performance Measurement is the
description of expected technical achievement. Actual project progress is compared
using fine–grained measurements and tests. The difference between the planned
progress and the actual progress represents a technical variance.
The basis of measuring software development progress is “testable requirements.”
[18, 19] A testable requirement defines the condition or capability for a software
system to meet its objectives.
6

7. Testable requirements follow the SMART acronym: Specific, Measurable, Achievable,
Relevant, and Time–Based. The criteria are met only if it is possible to write a test
case that validates that the requirement has or has not been implemented correctly.
The Software Management “Level of Effort” Problem
Traditional methods of comparing actual spending to planned spending are
inadequate for establishing, assessing, monitoring, and predicting the future
performance of a software project. The traditional approach of comparing “budget” to
“costs” fails to consider the technical achievements – the “physical progress” – that
is accrued for the software development tasks. In the traditional software
development project, the passage of time is assumed to be equivalent to progress
toward the completion. Software projects all too often have no tangible deliverables,
which are physically visible. The common outcome is that large costs are incurred
with little useful product being delivered until the end.
The first step was to identify the “value” of software deliverables as a set of
“verifiable” outcomes. These outcomes have two attributes critical to the deployment
of EV:
 They are defined in fine–grained self–contained units of work. [3]
 Each unit of work is testable to assure the requirements can be verified on fine–
grained boundaries. [14]
The EV principles are then applied to these attributes:
 Plan all work scope for the project through completion at a level sufficient to
measure progress through testable requirements. [4]
 Integrate project work scope, schedule, and cost objectives into a performance
measurement baseline plan against which accomplishments may be measured.
 Use actual costs incurred in accomplishing the work performed.
 Objectively assess accomplishments at the work performance level using testable
requirements.
 Analyze significant variances from the plan, forecast impacts, and prepare an
estimate at completion based on performance to date and work to be performed.
Project Selection Process
With these three processes in place (BSC, EV, and Testable Requirements) deciding
which projects will receive funding, which ones have the highest payback, and which
ones to concentrate on turns out to be an ill–structured problem. [4, 11] Addressing
this problem requires three activities:
 Representing the problem by uncovering information and the supporting
justifications for the project.
 Stating the solution to the problem by gathering options and selecting among
them.
 Supporting the justification through measures and tests.
3
Defining deliverables on fine–grained boundaries (1 to 3 days) increases the resolution of the
performance metrics as well as the accuracy of the Estimate at Completion (EAC).
4
In our method this level of detail is performed for an iteration. The completion of the project is planned,
but this level of detail is focused on the current and next iteration.
7

8. Since the selection problem has multiple solutions gaining agreement on which
projects to include requires considerable effort. In order to succeed, three actions
guide the project selection process:
 Define of the strategies that identify the value of the proposed projects.
 Make use of formal evaluation principles.
 Define the process by which projects are evaluated for their contribution to the
strategic goals.
There are many selection and evaluation algorithms, some simple, some complex.
We have chosen two simple approaches:
 Make a list – construct a prioritized list of the projects in order of importance.
This seems “too simple” to actually work. If the priorities on the list are the
implementation priorities of the objectives in the strategy map, then it is simple
and it works.
 Paired Comparison Analysis – for projects whose priorities are difficult to sort out
or ones that have other conflicts that a simple linear list cannot deal with, Paired
Comparison Analysis is a useful tool. It is a list making algorithm, but one to sort
out the priorities be asking “between these two projects, which one must come
first?” Paired Comparison Analysis helps work out the importance of a number of
options relative to each other particularly when there is no objective data. [4, 11]
Critical Success Factors for Portfolio Management
Using the four processes defined in Figure 2, Project Portfolio Management is driven
by the following critical success factors:
 Process – define a clear and concise process by which projects are categorized,
evaluated, and selected.
 Strategy – every project must have a “strategic” purpose and support an
objective in the balanced scorecard. If not it should be dropped
 Relationships – collections of projects are difficult to define in the absence of a
higher–level mission. The Balanced Scorecard provides the means of organizing
projects into initiatives. These initiatives can then be directly traced to Strategy
Map objectives.
 Decision framework – deciding which projects are to be executed first can be
answered by asking which projects need to be executed to fulfill the strategic
objectives in the proper sequence.
 Decision framing – clearly define the problem to be solved, the decision criteria,
the realistic tradeoffs and options.
References
1. Ben–Menachem, Mordechai and Roy Gelbard, “Integrated IT Management
Toolkit,” Communications of the ACM, April 2002, 45(4), pp. 96–102.
2. Berkman, Eric, “How to use balanced scorecard,” CIO, May 15, 2002.
3. Boehm, Barry and Kevin Sullivan, “Software Economics: a Roadmap,”
Proceedings on the Future of Software Engineering, Limerick Ireland, 2000.
4. Clemen, Robert T., Making Hard Decisions: An Introduction to Decision
Analysis, 2nd Edition, Duxbury Press, 1995.
8

9. 5. Cobbold, I. M. and G. J. G. Lawrie, “The Development of Balanced Scorecard as
a Strategic Management Tool,” 2GC Active Management, Maidenhead, England.
6. Constantine, Larry, “Work Organization: Paradigm for Project Management and
Organization,” Communications of the ACM, October 1993, 36(10), pp. 35–43.
7. Covey, Stephan, The 7 Habits of Highly Effective People, Simon & Schuster,
1990
8. Ferraro, Mike, “Technical Performance Measurement – A Program Manager’s
Barometer,” Program Manager, November/December 2000, pp, 14–20.
9. Fleming, Quentin and Joel Koppelman, Earned Value Project Management, PMI,
2002.
10. Kaplan, Robert S. and David P. Norton, The Strategy Focused Organization,
Harvard Business School Press, 2000
11. Mollaghasemi, Mansooreh and Julia Pet–Edwards, Making Multiple–Objective
Decisions, IEEE Computer Society Technical Briefing, IEEE Computer Society,
1997.
12. Niven, Paul R., Balanced Scorecard Step–By–Step: Maximizing Performance
and Maintaining Results, John Wiley & Sons, 2002.
13. Pisano, N. D. Commander, USN, “Technical Performance Measurement, Earned
Value, and Risk Management: An Integrated Diagnostic Tool for Program
Management.”
14. Sanderson, Sandra A., “Earned Value (EV) Management Model for Small
Software Development Projects,” Navy Mission Planning, 2001 Software
Technology Conference.
15. Scott, Judy E. and Iris Vessey, “Managing Risks In Enterprise Systems
Implementations,” Communications of the ACM, April 2002, 45(4), pp. 74–81.
16. Solomon, Melissa, “Project Portfolio Management,” Computer World, March
2002.
17. Summer, Mary, “Critical Success Factors in Enterprise Wide Information
Management Projects,” SIGCPR, 1999.
18. Wilson, P. B., “Testable Requirements – An Alternate Sizing Measure,” The
Journal of the Quality Assurance Institute (October 1995):1
19. Wilson, P. B., “Sizing Software with Testable Requirements,” Systems
Development Management, 34–10–04, Auerbach Publications, 2000.
Glen B. Alleman
Director, Program Management Office
Information and Network Services
Communications Group
CH2M HILL
Rocky Flats Environmental Technology Site
Golden, Colorado
9

10. 5. Cobbold, I. M. and G. J. G. Lawrie, “The Development of Balanced Scorecard as
a Strategic Management Tool,” 2GC Active Management, Maidenhead, England.
6. Constantine, Larry, “Work Organization: Paradigm for Project Management and
Organization,” Communications of the ACM, October 1993, 36(10), pp. 35–43.
7. Covey, Stephan, The 7 Habits of Highly Effective People, Simon & Schuster,
1990
8. Ferraro, Mike, “Technical Performance Measurement – A Program Manager’s
Barometer,” Program Manager, November/December 2000, pp, 14–20.
9. Fleming, Quentin and Joel Koppelman, Earned Value Project Management, PMI,
2002.
10. Kaplan, Robert S. and David P. Norton, The Strategy Focused Organization,
Harvard Business School Press, 2000
11. Mollaghasemi, Mansooreh and Julia Pet–Edwards, Making Multiple–Objective
Decisions, IEEE Computer Society Technical Briefing, IEEE Computer Society,
1997.
12. Niven, Paul R., Balanced Scorecard Step–By–Step: Maximizing Performance
and Maintaining Results, John Wiley & Sons, 2002.
13. Pisano, N. D. Commander, USN, “Technical Performance Measurement, Earned
Value, and Risk Management: An Integrated Diagnostic Tool for Program
Management.”
14. Sanderson, Sandra A., “Earned Value (EV) Management Model for Small
Software Development Projects,” Navy Mission Planning, 2001 Software
Technology Conference.
15. Scott, Judy E. and Iris Vessey, “Managing Risks In Enterprise Systems
Implementations,” Communications of the ACM, April 2002, 45(4), pp. 74–81.
16. Solomon, Melissa, “Project Portfolio Management,” Computer World, March
2002.
17. Summer, Mary, “Critical Success Factors in Enterprise Wide Information
Management Projects,” SIGCPR, 1999.
18. Wilson, P. B., “Testable Requirements – An Alternate Sizing Measure,” The
Journal of the Quality Assurance Institute (October 1995):1
19. Wilson, P. B., “Sizing Software with Testable Requirements,” Systems
Development Management, 34–10–04, Auerbach Publications, 2000.
Glen B. Alleman
Director, Program Management Office
Information and Network Services
Communications Group
CH2M HILL
Rocky Flats Environmental Technology Site
Golden, Colorado
9