IPM 2009 presentation. The Performance Measurement Baseline is the collection of the Cost, Schedule, and Technical Performance Measures for the program - used to make management decisions.

1. Thank you for the opportunity to speak today about an important topic.
This conference is primarily about applying Earned Value to programs in an attempt to
manage their performance. To keep them on schedule, control their costs, and assure that
the deliverables are compliant with technical performance requirements.
But having EV correctly performance its role is a necessary but not sufficient condition fro
success.
Starting with a “credible” Performance Measurement Baseline (PMB) moves us toward the
St ti ith “ dibl ” P f M t B li (PMB) t d th
sufficient condition.
1

2. I’ll present today eight (8) core steps in developing a credible Performance Measurement
Baseline (PMB).
These appear to be “easy” steps in principle. In practice of course all things are hard when
it comes to successfully managing programs.
But let’s start with the principles. And let’s also remember Yogi Berra’s quote
In theory there is no difference between theory and practice. In practice there is.
So let’s talk a little bit about the theory, but focus our time here on the practice of building
a credible Performance Measurement Baseline.
Glen B. Alleman
Lewis & Fowler, 8310 South Valley
Highway, Englewood CO80112
www.lewisandfowler.com 2

3. This may be a new concept, this is the phrase Probability of Project Success (PoPS), will be
our guiding light here. If the processes described here don’t increase our Probability of
Success, then
You may assume that all projects are successful, but we all know that’s not the case.
There are many reports about troubled projects. In fact some firms make their living by
reporting how bad projects are. It is assumed then that your project will be troubled as
well.
But let’s invert that notion and look at project management from another point of view.
How do we increase the probability of success of our project? We’ve been told our project
is not doing well, but how can we improve?
There are several important words here:
Increase make better.
Increase – make better
Probability – the statistical chance that something will occur.
Success – the beneficial outcome of some kind of effort.
So if we want to increase the probability of our project’s success what do we have to do?
But there’s one word that keeps recurring – DONE.
It’s really DONE that we’re after. That’s what the customer bought, that what the customer
wants to happen from your project.
3

4. The existence of the Performance Measurement Baseline (PMB) is necessary but not
sufficient for program success. In the EVMIG there is mention of the budget plan for
accomplishing the work.
The first question is ‐ “what work?”
If we start with a Systems Engineering paradigm, the core measurements of Measures of
Effectiveness and Measures of Performance
MOE ‐ Operational measures of success that are closely related to the achievement of
MOE O ti l f th t l l l t d t th hi t f
the mission or operational objective being evaluated, in the intended operational
environment under a specified set of conditions: (1) stated from the customer point of
view; (2) focused on the most critical mission performance needs; (3) independent of any
particular solution; (4) actual measures at the end of development.
MOP ‐ A criterion used to assess friendly actions that is tied to measuring task
accomplishment. Measures that characterize physical or functional attribute relating to
accomplishment Measures that characterize physical or functional attribute relating to
the system operation: (1) supplier’s point of view; (2) Measured under specified testing
or operational conditions; (3) assesses delivered solution performance against critical
system level specified requirements; (4) risk indicators that are monitored progressively.
4

5. Building a credible PMB starts with identifying the architecture of the IMP and the
supporting tasks in the IMS.
Although this is restating the obvious the process to do this is actually quite hard.
Adding schedule and cost risk identification and mitigation to the process is the
minimal result for a winning proposal.
It cannot be emphasized enough – the architecture of the IMS is critical to
identifying i k t l
id tif i a risk tolerant schedule. The “rats nest” approach i simply unacceptable
t h d l Th “ t t” h is i l t bl
to the success of any program.

6. The term “credible” is important. Credible is not the same as accurate, precise, or words
like that. Credible means it is a trustworthy measure – believable.
There are other definitions, but each should support the notion that the information
provided in the PMB is a trusted basis of decision making.
This means if I make a management decision on this data, I won’t be disappointed late to
find out the data was not correct.
6

7. This list comes from guidance in the defense program management business. While some
of the statement may not be appropriate for the commercial world, every statement has
some applicability to every project – not matter what the domain. Whether formal or
informal the programmatic risk assessment of a project needs to ask or answer these
statements.
The actions needed to close any gaps from these statements are outside the scope of this
presentation.
But the next step is to have the project management team start to answer these questions.
7

8. Here’s the top level elements of the Performance Measurement Baseline.
The customer bought a product. What is the topological arrangement of that product in
the form of a Product Tree. MIL‐STD‐881A introduces the notion of a Product Tree as the
basis of a credible Work Breakdown Structure
The teams that are going top produce the product must be defined. The artifact of this
definition is the Responsibility Assignment Matrix
The Statement of Work foot and ties the products with the deliverables
Th St t t f W k f t d ti th d t ith th d li bl
The cost “basis of estimate” is derived directly from the work definition at the Work
Package level
The Integrated Master Plan defines how the program strategy will be executed.
The Integrated Master Schedule defines the sequence of work that must be performed to
implement the strategy.
implement the strategy
8

9. Here’s our 8 steps to building the Performance Measurement Baseline.
Each step should be performed in the order defined here.
1. Define the product or service to be delivered
2. Define the maturity assessment points – the places where the pre‐defined technical
performance measures will be assessed
3. Define the accomplishments needed to move the product to a pre‐defined level of
maturity
4. Define the measurement criteria needed to performance the maturity assessment
measurements. These are the “exit” criteria for the individual work packages.
5. Define those work packages. Keep them inside a duration that answers the question
“how long are you willing to wait before you find out you’re late?” Most firms limit the
work packages to 60 calendar days
work packages to 60 calendar days
6. Connect the work packages in a sequence to provides increasing technical maturity and
minimal exposure to programmatic and technical risk.
7. Assemble all this information into an Integrated Master Schedule
8. Do this integration in an incremental manner – one program event at a time.
9

10. Here are the primary “parts” of the Performance Measurement Baseline (PMB) and the
order in which they are developed and used by subsequent “parts.”
The “parts” shown here are the macro level elements of the PMB. The details of their
individual construction, content, semantics, and syntax are outside this presentation.
The key concept here is that the PMB flows “down” starting with the WBS/CWBS. This
concept is repeated over and over in this presentation. It cannot be said enough – no
credible WBS, no credible PMB.
One critical element in this process is the WBS Dictionary. This is a description of the work
products of the WBS elements. There are many examples of good WBS dictionary on the
web. Google will find them for you, even an avionics system WBS dictionary. Use these
example to develop the OH‐58 dictionary, check it against know working examples, and
improve the current WBS dictionary from these examples.
The dictionary shows the hierarchical relationship of the elements and describes each WBS
The dictionary shows the hierarchical relationship of the elements and describes each WBS
element and the resources and processes required to produce it. It also provides a link to
the detailed technical definition documents. The WBS dictionary should be routinely
revised to incorporate changes and should reflect the current status of the program
throughout the program's life. – from §2.2.3.1 of MIL‐STD‐881A
10

11. There are actually three performance measurement baselines that need to be established.
1. The Technical Baseline – this is what is being built by the program.
2. The Schedule Baseline – this is how we’re going to build it
3. The Cost Baseline – this is how much it will cost to build what we need to build
11

12. So let’s try out these ideas on a semi‐real enviornment
12

13. Here’s a picture of actual output a capabilities development session for a major ERP system
integration program.
Starting in the upper left, the needed capabilities are captured through a Product
Development Kaizen with the stakeholders in an offsite session. These Kaizens are full
contact meetings where the participants work “on the wall” to reveal the capabilities and
the attributes of these capabilities.
These sticky notes are then captured in some organizing tool. My favorite for this level of
detail is MindJet’s Mind Manager. It is a hierarchical organizing tool that can export its
structure to MSFT Project.
No matter what tool you use, you’ve got to have some way to “discover” the business
capabilities before moving the next stage of the project.
Without a clear and concise description of the needed capabilities you’ll have a hard time
recognizing Done when it arrives if it ever arrives.
recognizing “Done” when it arrives – if it ever arrives
We’ve all heard of, or possibly used a system that met all the requirements but failed to
provide the business value that was promised.
The development of the Capabilities – preferably in a Concept of Operations document is
the foundation for the remaining efforts in increasing the probability of success for your
project.
With the Capabilities in hand, the technical and operational requirements become clear –
or at least clearer.
13

14. The elements of the IMP and IMS along with the cost and technical performance measures
are driven by the following sources of data.
The Statement of Work is the starting point
The WBS/CWBS describes the product structure
Task in work packages show the work activities
The CDRLs guide the deliverables at least through PDR
Accomplishment Criteria are the exist criteria for the Work Packages
Significant Accomplishments collect the work products needed to move the program to
the next level of maturity
The Program Events are the assessment of this maturity
When someone asks why are we doing this – the answer is here
y g
14

15. When we speak about increasing maturity, evolutionary development and measures of
physical percent complete – one good way to visualize this is through a product maturity
flow diagram.
This diagram is a “real” one from a claims processing ERP system rollout, complete with
COTS (Commercial Off The Shelf) integration with legacy systems and custom software
development. The worse of both worlds.
The critical success factor here is the define upfront what done looks like for all the
The critical success factor here is the define upfront what “done” looks like for all the
incremental business value elements of the program.
Each of the circles is a point of maturity, where the delivered product or service can be put
to use in some way.
As well the project can be canceled at each of these and the investment put to work. Many
people would be unhappy, but the financial aspects of the program would remain intact.
This type of diagram also shows the dependencies between the down stream (right) and up
stream (left) elements of the project. What has to come first, before business value can be
delivered.
In other domains (defense and government) these diagrams are built during the Product
Development Kaizen process to show what must be done to get to “done.”
15

16. Now let’s put all this together.
Starting with the WBS, the terminal nodes are represented in project with Work Packages.
You cannot have a project without some form of a Work Breakdown Structure (WBS). Building a
good WBS is a day long workshop all in itself.
So how many here have a Work Breakdown Structure for their project?
This may not be a term you’ve heard of before. A Work Package is a lump of work that produces a
single outcome. A “package of work” that produces something.
For the schedule putting the Work Packages in the proper order is the starting point for a credible
For the schedule putting the Work Packages in the proper order is the starting point for a credible
schedule. If you can get the Work Packages in the right order, with their durations defined, then
you have the start of the credible schedule.
The next step is to not do nay more scheduling in MSFT project. Instead let the Work Package
manager look have the activities inside the Work Package and be done with it.
This is how large Defense and Space programs schedule. There are three levels of schedules
mandated by a government “rule,” DID 81650.
For the commercial side there is no mandated regulation. But 881A and the PMI Work Breakdown
g
Structure Guide are the best places to start.
The Master schedule – a top level “picture” what is happening in what order.
The Intermediate Schedule – the sequence of Work Packages.
The Detailed Schedule – the day to day activities of the project.
For IT projects, the Intermediate schedule is a sweet spot. One that connects cost with work and
deliverables. One that minimally imposes effort on the team and fits well with the agile world of
Corporate IT software development.
Corporate IT software development
Since agile is focused on defining deliverables and measuring progress through working software, it
is a natural fit for a WBS.
16

17. Now let’s put all this together.
Starting with the WBS, the terminal nodes are represented in project with Work Packages.
You cannot have a project without some form of a Work Breakdown Structure (WBS). Building a
good WBS is a day long workshop all in itself.
So how many here have a Work Breakdown Structure for their project?
This may not be a term you’ve heard of before. A Work Package is a lump of work that produces a
single outcome. A “package of work” that produces something.
For the schedule putting the Work Packages in the proper order is the starting point for a credible
For the schedule putting the Work Packages in the proper order is the starting point for a credible
schedule. If you can get the Work Packages in the right order, with their durations defined, then
you have the start of the credible schedule.
The next step is to not do nay more scheduling in MSFT project. Instead let the Work Package
manager look have the activities inside the Work Package and be done with it.
This is how large Defense and Space programs schedule. There are three levels of schedules
mandated by a government “rule,” DID 81650.
For the commercial side there is no mandated regulation. But 881A and the PMI Work Breakdown
g
Structure Guide are the best places to start.
The Master schedule – a top level “picture” what is happening in what order.
The Intermediate Schedule – the sequence of Work Packages.
The Detailed Schedule – the day to day activities of the project.
For IT projects, the Intermediate schedule is a sweet spot. One that connects cost with work and
deliverables. One that minimally imposes effort on the team and fits well with the agile world of
Corporate IT software development.
Corporate IT software development
Since agile is focused on defining deliverables and measuring progress through working software, it
is a natural fit for a WBS.
17

18. The collection of work packages, their relationship to the “exit criteria,” and then to the
Significant Accomplishments landing on the Program Event is the topology of the
Performance Measurement baseline – once the basis of estimate and the Technical
Performance Measures are added.
18

19. A critical success factor for any program is to make visible the flow of “increasing maturity”
for each deliverable. It is necessary to show how the work efforts are sequenced to
produce the deliverables. In the absence of the measure of “increasing maturity” these
work efforts have not units of measure.
The result is progress is measured as the passage of time and consumption of resources.
What is needed is to measure progress in units of planned compliance with the Technical
Performance Measures. Not a few TPMs for the program as a whole. But a planned
measure of compliance for each Work Package outcome.
This approach assures that a description of what “done” looks like is available for each
work effort.
19

20. Here’s a sample of a Microsoft Project file that models the structure shown a few slides
back.
20

21. One of the new ideas about building the PMB is to only connect the Work Packages
together – instead of the Tasks. Tasks within the Work Package can be
interconnected.
But between the Work Packages, if there is interconnections the work on the right
will be using partially completed work on the left.
If this is done, then the work package on the right will absorb this partially
completed work, and when the work on the left is completed, re-work may be
re work
needed of anything changed.
Treat the work packages as complete “lumps of work,” with 100% fidelity to the
planned deliverable.
This way the work package on the right can start with fully formed raw materials.
21

22. The same approach should be taken for the Significant Accomplishments and the Program
Events.
Partially completed work should not cross a Program Event boundary unless it is long lead.
Otherwise you are moving to the future “re work” from the past.
22

23. Here’s an example of how this looks in a real project, using the PERTChart Expert tool from
www.criticaltools.con
23

24. The management of risk is how Adults manage projects – Tim Lister
The Integrated Master Schedule must show how risk is being “retired” if that schedule is to
be considered credible.
Without this description, when the risk turns into an issue, there will be little time to make
the correction and mist likely the reserve assigned to the risk will have been consumed.
Risk Retirement is better than Risk Mitigation.
24

25. The core elements of any program – cost, schedule, and technical performance measures –
are random variables. They have variance and standard deviations. These statistical
behaviors are drawn from an underlying probability distribution.
Without the understand of the statistical nature of the cost, schedule, and technical
performance measures, the information produced during the program’s execution is not
only untrustworthy, it is not credible.
25

26. To produce a Credible Performance Measurement baseline we must do more than
announce we have a “credible performance measurement baseline,” we must actually
demonstrate this credibility in units of measure meaningful to the customer.
26

27. When we say “credible” what do we really mean?
What are the units of measure of credible?
27

28. If we don’t understand the underlying statistics and more importantly that these statistics
drive the probabilistic behavior of the plan, the related costs and the technical performance
as well, then we are working in the dark.
Remember the previous discussion of risk and credibility.
Knowing the underlying variances from the underlying probability distribution functions is
the basis of credibility.
Without this information only a 50/50 guess is possible.
With t thi i f ti l 50/50 i ibl
28

29. Analysis of Probabilistic Schedule and Last Updated: 11/2/2009
Cost Models
When we speak of probabilistic risk analysis, we also need to speak of the statistical
nature of the activity network.
When we speak of a probabilistic activity network (a Bayesian network) we also
need to speak in terms of probability.
A question that can be asked of the network is – “ what is the probability of
completing this task by a certain date?”
A second question th t can be asked i – “ h t are th underlying statistics of th
d ti that b k d is “what the d l i t ti ti f the
activities of the network?”
A final question that needs to be asked is “what is the inherent uncertainty in these
estimates?”
In other words – how good is our ability to guess in the presence of a statistical
process?
Prepared by Glen B. Alleman
December 2005 29/176

30. So now we’re back to the problem of measuring credibility in units meaningful to the
customer.
Here’s the beginning of this measurement.
1. Do we know the underlying statistical behavior of the cost, schedule, and technical
performance measures?
2. Do we have a credible architecture for producing and assessing the increasing maturity
of the products or services? Not just assessing but also producing.
f th d t i ?N tj t i b t l d i
3. Can we define what “done” looks like in measures of effectiveness and measures of
performance?
30

31. So what happened to all the Earned Value stuff. That’s what we are here for Earned Value.
It’s here, but now it has new meaning.
Actually it has a different meaning.
Earned Value reports performance from the past.
What we want of forecasts of the performance in the future.
31

32. And we need to connect that forecast of future performance to the technical performance
of product.
f d
Being on schedule and on budget but being out of compliance with the technical
performance measures is not good. More work is now needed to get the product “on
spec.”
Some might say that “off spec” is recognizable. But if the performance specification (MoP
or MoE) are statistical compliance, it’s not that straight forward.
We must understand what it means to be statistically compliant. This is a statistical process
control issue. In what Standard Deviation are we compliant. It’s different to be in the 1st
standard deviation than in the 2nd standard deviation.
This presentation is too short to dive into the issues around probabilistic compliance
models, but remember is simple story
What is the most likely (the mode) of temperatures in Trinidad Tobago and Cody
Wyoming?
Wyoming?
Well they are the same 78°
What is the variance?
Well that’s different
In Trinidad its about ± 5°
In Cody its – 50 to +20
In Cody its 50° to +20°
32

33. So now we can understand that we need all three measures
Cost, schedule, and technical performance to establish the basis of credibility.
Each measure must be statistically adjusted for its particular probability distribution.
Each measure must have a model of how it interacts with all the other measures.
Each measure must describe the work processes that are used to increase the maturity and
therefore the probability of program success.
A simple static model the program is not credible. A dynamic model is the beginning of this
credibility.
33

34. 34