Too many organizations today still measure the success of a project based only on the traditional project management standards of delivering On Time, On Budget and On Scope. While these criteria are valid measures of successful project management, they are less suitable when assessing a project’s true success: its contribution to the overall organization's performance. Indeed, the ulti-mate success of a project – whether cost savings, revenue increases or customer satisfaction improvements – may not be known until years after it has been successfully delivered.

1. Optimizing Organizational Performance
by Managing Project Benefits
A UMT White Paper
By Yorai Linenberg, Ziv Stadler, and
Stephen Arbuthnot

2. <2>
Introduction
Too many organizations today still measure the success of a project based only on the traditional project management standards of
delivering On Time, On Budget and On Scope. While these criteria are valid measures of successful project management, they are
less suitable when assessing a project’s true success: its contribution to the overall organization's performance. Indeed, the ulti-
mate success of a project – whether cost savings, revenue increases or customer satisfaction improvements – may not be known
until years after it has been successfully delivered.
The challenge senior IT and business managers face today, however, is not only to successfully manage the project throughout its
lifecycle, but to quantify upfront how a project’s success, or the benefits it generates, can be measured. Once these benefits have
been committed to, they need to be tracked throughout the post implementation phase in order to ensure that the project pro-
vides the greatest strategic value to the organization. By identifying and remaining cognizant of a project’s benefits, organizations
can make great strides towards ensuring they are maximizing its ultimate success.
Optimizing an organization's performance, however, needs to begin long before individual projects’ benefits are defined, their exe-
cution managed, and their delivery tracked. Rather, the organization's decision-makers must first ensure that they have chosen the
most appropriate projects for inclusion in the portfolio. Even the most successful project delivery will not help to optimize the or-
ganization's overall performance if the projects chosen for the portfolio are not aligned to the business strategy.
A project benefits management framework should therefore be viewed
as an essential step towards improving the efficiency of a portfolio of
projects, and ensuring that the expected benefits are aligned with the
business strategy, and in fact, being realized. Such a framework natu-
rally complements, and should be integrated within, an overall project
portfolio management (PPM) approach, which views a portfolio of pro-
jects in the same way as a portfolio of investments. The ultimate, long
term objective of both is obviously to increase the portfolio owner’s
(whether an individual or organization) return on investment (ROI).
There are three main differences, however, between these portfolio
types, which explain why such a framework is so important:
1. In the short term, it is not always possible to measure a project’s
financial contribution; therefore additional measurable benefits
must be identified.
2. Whilst the performance of a financial portfolio can be measured at
any point in time, the success of a project, and its contribution to
the organization's strategy, can often only be measured long after
it has been delivered.
3. While the contribution of each security to the overall performance
of a financial portfolio can be readily identified, due to inter-
project dependencies and overlaps in scope, a measurable organi-
zation benefit is usually the outcome of several projects.
By applying the same units of measurement to both the organization's measurable objectives and the individual projects’ benefits,
the steps involved in implementing a benefits management framework create a visible link between the two. At a high level these
steps include:
(1) Formulating and prioritizing the organization's objectives, and defining their relevant key performance indicators (KPIs) and
targets; (2) Identifying each individual project’s benefits and determining which ones are best positioned to contribute to the ob-
jectives’ KPI target; (3) Rationalizing and optimizing the project portfolio to ensure the aggregated project benefits achieve the
objectives’ target KPI; (4) Sequencing the optimized project portfolio to generate the benefits realization timeline, taking into ac-
count resource constraints, dependencies and critical milestones; (5) Managing scope and timeline changes throughout the pro-
ject’s execution and evaluating their impact on the benefits and their delivery timeline; and (6) Monitoring the project benefits
Business’ Desired Objectives
AbilitytochoosetheRightProjects
100%
80% - Choosing the best projects X
75% - Achieving the desired benefits
60% - Business’ Realized Objectives
100%
Exhibit 1 – Doing the Right Projects Right
Recent findings indicate that organizations did not achieve
their targets because many projects were not aligned with the
business objectives, and of those that were, a high portion did
not realize their intended benefits. The Standish Group, 2001

3. <3>
realization phase, which usually stretches beyond the project’s “official” end date, to verify project success and ensure the organi-
zation's objectives are being achieved.
Step 1: Formulating the Organization's Objectives
An organization's first step towards optimizing the performance of its portfolio of projects is to identify and define its key objec-
tives. Objectives must to be specific in scope, action-oriented, able to serve as high level goals for an individual project, and should
ideally have been agreed upon through a consensus approach by as wide a group of senior managers and stakeholders as realisti-
cally possible. Objectives are typically categorized into three areas: demand management (e.g. sales effectiveness, increased reve-
nue), supply management (e.g. operational efficiency, customer responsiveness) or support services (e.g. infrastructure, regulatory
requirements). Most importantly, however, objectives need to be measurable: each one should have a set of one or more KPIs
which, along with their defined targets, will enable decision makers to monitor and control the successful delivery of the objectives
in real terms.
KPIs: Measuring Objectives in Real Terms
Defining the “right” KPIs is one of the most difficult tasks an organization faces when establishing a benefits management frame-
work. Effective KPIs must be relevant to their associated objective, specific, readily measurable by the organization, realistic and
achievable, and contain a time element. That a KPI must be relevant to the objective it is striving to measure may seem self-
evident, but this point still needs to be highlighted. It can often be convenient for an organization to employ an existing perfor-
mance metric to measure the impact on a recently defined objective. If the price of this convenience, however, is the undermining
of the accurate measurement of the achievement of the objective, then a new KPI should be chosen.
The KPI also needs to be specific enough so that it can measure the delivery of an individual objective, without being excessively
affected by external noise and unrelated events. For example, how can a KPI as broad as a customer satisfaction index measure the
direct impact of a specific project? Clearly this is when the science becomes less exacting, but it highlights the effort needed to en-
sure the KPIs are well defined.
That a KPI must be measurable is often overlooked in the early stages of such an exercise. It may seem obvious that the best meas-
ure of increased customer satisfaction should be a higher score on a customer satisfaction survey, however if no such metric is cur-
rently being tracked, that KPI may not be the most appropriate. A better KPI might be to measure the number of calls to a help line
over a specific period. A side effect of a PPM exercise, however, should not be a host of new projects whose sole purpose is to de-
velop processes to measure the success of other projects!
As with any goal, the target KPI must be realistic and achievable for it to be relevant. There is no point in setting up targets which
are beyond the organization's, or, for that matter, the market’s ability to fulfill.
The final requisite characteristic of a KPI is that it contains a time element. In simple terms, the KPI must measure not only how
well the organization is doing in terms of achieving its objectives, but it must also provide the time dimension for achieving these
objectives.
Prioritizing the Organization's Objectives
Once the objectives have been agreed and the KPI’s defined, and accepting that for a given period (e.g., 12 or 18 months) not all
objectives may carry the same importance, a prioritization exercise must next occur. For example, after analyzing market condi-
tions, an organization may wish to focus its efforts (and initiate projects) to reduce its cost-base, rather than to increase its reve-
nues.
“Portfolio management takes a holistic view of a company's overall IT strategy. Both IT and business leaders vet
project proposals by matching them with the company's strategic objectives.”
CIO.com, Portfolio Management - How to do it Right

4. <4>
The prioritization process is therefore used to determine the relative importance of the objectives and to generate a normalized
weights vector. UMT uses the Pairwise Comparison method, which ranks the objectives in pairs against one another, in order to
generate these values.
Such an exercise also has the additional advantage of creating transparency and achieving consensus among decision makers, by
forcing them to choose and explain their priorities. In some instances, however, the resulting discussion might be heavily depend-
ent on the organizational culture and level of openness.
Step 2: Identifying Project Benefits and Prioritizing the Portfolio
The KPIs described in the previous section are necessarily closely linked to project benefits. Whereas a KPI measures progress to-
wards achieving an objective, a benefit is the measurable, positive outcome of a project that a KPI is measuring. Certainly projects
will also provide benefits that are not measured by one of the KPIs, but when analyzing projects from a portfolio view, in the con-
text of an organization's objectives, we should only be concerned with the benefits that can be measured by these KPIs.
The challenge is therefore to create a mechanism that will link, using the same units of measurement, the individual project’s bene-
fits to the objectives’ KPIs. Such a framework will not only enable the organization to pinpoint potential gaps in its portfolio (where
the consolidated projects’ benefits do not add up to the objectives’ target KPI), but will also provide a way of prioritizing the entire
project portfolio against its objectives.
Linking Projects to Objectives
Similar to agreeing on the relative weighting of objectives, a consensus approach should be used to ensure that all the key stake-
holders agree on the contribution each project makes towards achieving each objective. Therefore, once each proposed project
has been defined – including an estimate of cost (both by resource-type and monetary measure), scope, schedule, dependency,
and other potential characteristics specific to the organization – it must then be appraised based on the contribution, if any, that its
benefits make towards the achievement of each objective. Examples for project benefits are “reducing headcount in the ac-
counting department by two (2) FTEs”, or “reduce error rate in transaction processing by 10%”.
In creating such a mechanism, it is necessary to establish KPI bands, or threshold levels, that will enable project sponsors to deter-
mine how strongly their project supports each one of the objectives and their KPIs. UMT uses a structured approach, usually as part
of a redesigned portfolio planning or project approval process, for defining these bands and assessing the contribution of each pro-
ject. The KPI threshold levels are translated into numerical bands (usually 5), where “Extreme” might mean that a project will con-
tribute 10%-12% of the target KPI, “Strong” 8%-10%, etc. The definition of these bands usually depends on the number of projects
that can show a link to a given KPI.
Evaluate the Relative Importance of Each
Business Objective
Exhibit 2 - Prioritizing the Organization's Objectives
Review and Agree on the Business Objectives and
Priorities

5. <4>
Prioritizing Projects: The Project’s ‘Strategic Value’
Once each project has been evaluated against the objectives and their KPIs, it can be assigned a “Strategic Value”. This value repre-
sents the level of support a project is expected to provide to the organization's objectives, and is calculated using (a) the weights of
the relevant objectives the project is supporting and (b) the “strength” of this support (i.e., Extreme, Strong, etc.). The strategic
value may then be used as a proxy for the priority of the project within the organization's portfolio.
It is important to note that, when using this approach within a project portfolio management framework, such a methodical pro-
cess generally identifies project benefits that contribute to the achievement of as many of the organization's objectives as possible.
However, when organizations endeavor to conduct ad hoc project benefits management exercises independently from a PPM
framework, they often focus exclusively on financial benefits at the expense of ignoring others that also contribute to the overall
performance of the organization. It is generally accepted that over the long term, the primary justification for a project is its finan-
cial impact on the bottom line; however, since in the short term this impact is not always visible, other measurable project benefit
types must be considered. These may include non-financial (e.g. quality of service, workforce motivation and satisfaction, and man-
datory or legal requirements), or indirect ones (e.g. strategic fit, risk reduction, and internal management efficiency). Analyzing
project benefits within the context of a portfolio, therefore, serves as a powerful tool for organizations determined to optimize
their performance.
Step 3: Rationalizing and Optimizing the Project Portfolio
Having agreed on the organization's definition of performance, and having assessed how each project in the portfolio individually
contributes to this performance, the next challenge is to rationalize and optimize the project portfolio.
Rationalizing the portfolio is necessary in order to ensure it does not contain multiple projects that are conspicuously superfluous,
or others that are clearly mutually exclusive. Potential overlaps between projects should be exposed; for example, projects from
different business units may appear to achieve the same benefit, in the same functional area, by using different approaches. These
overlaps and duplications should be highlighted as alternatives or removed. This step obviously requires detailed knowledge and
understanding of the projects and their benefits.
Assess Each Project’s Contribution to the Overall
Target KPI
Prioritized Project List Based on
“Strategic Value”
Exhibit 3 – Linking Projects to Objectives
“Rather than trying to calculate specific returns on each project, some companies are saving millions
of dollars each year through a portfolio approach to IT spending.”
Computerworld, Stabilizing Your Risk

6. <4>
Once potential duplications have been dealt with, each category of benefits should be consolidated to evaluate whether the target
KPIs can be achieved. If gaps are identified, then decision must be made to either amend the scope of existing proposed projects or
propose new projects, in an effort to create a portfolio of projects that has the potential to achieve each objectives target level of
KPI.
Although the organization will strive to achieve all its target KPIs, in reality, this is not always achievable from an operational per-
spective. A portfolio optimization step will therefore be required, which will take into consideration the usual constraints faced by
IT managers: money, resources and their types, time, level of risk, etc. In a perfect world, with no constraints, the entire rational-
ized portfolio would be implemented, but in the real world, and especially in these economically challenging times, difficult deci-
sions need to be made in order to create the optimal project portfolio that delivers the maximum strategic value.
UMT uses various algorithms to create an optimal portfolio of projects that yields the greatest strategic value, no matter what con-
straints are applied. It is important to remember, however, that this optimized portfolio does not necessarily “select” the projects
with the individual highest strategic value, but rather it proposes the overall portfolio with the highest aggregate strategic value. It
also does not necessarily select projects with the greatest ROI, or net contribution to increased revenues – traditional characteris-
tics of approved projects – unless of course a heavily weighted objective’s KPI is increased revenue or ROI.
Once a project has earned its place in the optimized portfolio, following the prioritization and optimization steps, this phase’s final
step is for each project’s benefit realization schedule to be formally signed-off and committed.
The importance of this step can not be overstated, as it sets forth the commitments made by the project owners and managers
regarding the project’s delivery. This is also when the project’s benefits realization schedule – the manifestation of what the KPIs
will be measuring – is proposed, approved and therefore committed. The benefits realization schedule is what will be tracked dur-
ing a project’s execution and post-implementation phases to measure whether the benefits proposed during the approval phase
are indeed being achieved.
In the context of most organizations' traditional project lifecycle, this would often happen during the project approval process,
when a formal assessment is made to determine whether a project should be funded. While the level of documentation will vary
depending on the culture of the organization and the type of project, the underlying deliverable usually takes the form of a busi-
ness case, outlining both project costs and benefits.
Step 4: Sequencing the Optimal Portfolio
Once the optimal project portfolio has been selected, the next challenge is to create a roadmap for delivering the selected projects,
which, in turn, will define the overall benefit realization roadmap. Ideally, all selected projects could be fully staffed and kicked off
immediately. In reality, however, an organization's supply of resource types rarely matches the demand. Furthermore, additional
complications result from inter-project dependencies, external dependencies, critical milestones, etc.
A project sequencing analysis needs to be undertaken, therefore, to understand the resource gaps and the tradeoffs an organiza-
tion faces between different alternatives. For example, the costs of staffing up all projects based on the original timeline – where-
by expensive contractors or permanent staff may be hired during some busy project phases, which are then followed by extended
periods of resource underutilization - should be measured against the alternative of smoothing internal resource utilization volatili-
ty. This latter scenario may result, however, a delay to some projects, which will cause their specific benefits to be delayed; the
consequential knock-on effect will result in a delay in the achievement of the organization's overall target KPIs.
UMT uses dedicated software tools to enable IT managers to run “what-if” scenarios either to optimize the scheduling of projects
while keeping resource-type surpluses and shortages as flat as possible, or alternatively to keep a rigid schedule but identify clearly
expected resource gaps.
“More sophisticated analytical tools are offered by vendors such as United Management
Technologies… providing actionable measurements of variables such as cost, risk and resources to
help IT organizations maximize their IT Investments.”
Computerworld, Balancing the IT Portfolio

7. <4>
Step 5: Managing Changes throughout the Project’s Execution
While a project is being executed, there are a number of activities that must take place to ensure the portfolio’s project benefits
will indeed be realized. Firstly, each project’s approved benefits realization schedule – which was committed when the business
case was approved – must be validated at regular intervals, usually during the course of project status reporting. Regular, timely,
and effective project status reporting is a vital tool to provide early warning of an under-performing project, the effect of which
may likely impact the benefits realization schedule. More importantly, however, the impact of changes to the project must be fully
taken into consideration.
While conventional program management change control is a trademark of any well-managed program, it is commonplace for
scant attention to be paid to assessing how changes may impact project benefits. This is a significant oversight, however, since the
true success of a project cannot be measured without ensuring its proposed and signed-off benefits have been realized. Since a
project’s signed-off, expected benefits will be directly impacted by any changes to scope or timing, any changes to these two
parameters during execution need to be assessed through a formal change control process to identify the knock-on effect to the
overall benefits realization schedule.
Furthermore, if significant changes to a project do occur during the execution phase, and these have measurable impacts on the
benefits realization schedule – both magnitude of benefit achieved and timing of realization – corrective action should be consid-
ered by the project owner and stakeholders. If necessary, a decision to cancel the project may need to be made if it is determined
that organizational resources could be better deployed in such a way to maximize strategic value.
Finally, while changes to a project’s scope and timeline are generally within the organization's control, external events certainly are
not. If significant external events occur, they may have an impact on the organization's objectives, priorities and target KPIs and a
similar assessment of the project(s), within the context of the portfolio, should be conducted in order to determine what corrective
action may need to be made to ensure the expected benefits and their delivery timeframe are maintained. The assessment should
also take into account new projects that may need to be integrated into the current portfolio.
Step 6: Monitoring the Project Benefits Realization Phase
For most organizations, once a project has been implemented and the project team has been dismantled and redeployed across
new initiatives, the project is considered effectively closed. Many organizations wisely conduct a post-implementation review to
ensure all mistakes and shortcomings experienced during the project are documented and presumably avoided in the future, but
effectively no more attention is paid to the project.
This approach is a mistake. The success of a project, and hence its strategic value and contribution to overall organizational perfor-
mance, can only be truly declared once it has reached its Goal Achieved Status: the state when all the benefits identified in the
business case have been achieved. Only at this point, when the relevant KPIs indicate that the expected impact on the objectives
has been met, can a project be considered successfully completed.
Indeed, for many organizations it may be hard to identify the specific contribution a project has on the organization's performance.
Ideally, in order to track the project benefits during the post-implementation benefits realization phase, an organization should
have in place a process to clearly define the benefits and a process to collect the benefits data and report on them regularly both
by individual project, as well as for the overall portfolio, which in effect provides a scorecard of the organization's performance.
This final step in the benefits management framework provides a powerful report card view of the organization's actual perfor-
mance today, against the one that was proposed in the past when the original decisions were made. As such, difficult and often
contentious project investment decisions made in the past can be re-visited in the future and definitively evaluated for their judi-
ciousness. Good decisions are validated, while bad decisions are identified so that corrective action may be taken to ensure they
are not repeated.

8. Conclusion
The bad news is that any organization committed to optimizing its performance faces a myriad of options and will always be
forced to make difficult decisions. The good news, however, is that a holistic approach and a structured decision-making frame-
work help eliminate the bad options and illuminate the right decisions.
A framework for project portfolio management in general, and benefits management more specifically, brightens senior man-
agement’s decision-making spotlight and focuses it on the portfolio that delivers the mix of project benefits that brings the great-
est strategic value to the organization.
Key points to achieving success include:
 A delivered project – even though it may be on time, on scope, and even under-budget - does not equal a successful pro-
ject.
 Doing the right projects is as important doing the projects right.
 Strategic value, not ROI or NPV, is the currency of organizational performance.
 The only relevant project benefits are the ones that contribute to achieving organizational objectives.
 Decisions to select, change or kill a project must be made at the portfolio level; this is the only view that can see the im-
pact on the organization's performance.
 Rationalizing a portfolio can help save a lot of money by identifying duplications and overlaps; in parallel, this process also
highlights gaps which may result the organization's objectives not being achieved.
 A project’s benefits realization schedule is fickle: it needs to continually be revisited and validated because any little
change to the project will upset it.
 The Go Live date signals the start of the benefits realization phase, not the end of the project.
 Even a failed project can have some benefits; the same mistakes shouldn’t be made again.
References:
1. The Standish Group. (2001) CHAOS Report. West Yarmouth, MA
2. CIO.com (May 2003), Portfolio Management – How to do it Right, Todd Datz
3. Computerworld (May 2001); Stabilizing Your Risk, Robert L. Scheier
4. Computerworld (Feb. 2003 ) Balancing the IT Portfolio, Thomas Hoffman
# # #
1 Battery Park Plaza, 4th Floor, New York, NY 10004, (212) 965-0550, www.umt.com