1. Vendor Selection Decisions
4300 Wilson Boulevard, Suite 350
Ar l i n g t o n , V i r g i n i a 2 2 2 0 3
703-399-2100
www.decisionlens.com

2. Selecting the best vendor is a difficult challenge.
The stakes can be high, jobs may be on the line,
and, adding to this complexity, multiple stake-
holders from various factions within the organi-
zation may demand that their input and needs be
addressed throughout the selection process.
This white paper covers the Overview of Vendor Selection
Most organizations approach vendor selection (also called “source selection”) by
best-practice approach for
setting up an evaluation board representing the internal “customers.” The require-
vendor selection decisions ments are defined, and the vendors are asked to respond through an RFP. They are
and is meant for anyone then evaluated against a checklist of requirements by each evaluator, and sheets
with checkboxes showing “meets requirements” or “does not meet requirements” are
involved in a vendor
produced. The result is long lists of checkboxes at a detailed level, with little insight
selection, product selection, into whether the vendors are capable of delivering to the key criteria the organization
source selection, acquisi- must meet.
tion plan, procurement or Oftentimes evaluators aren’t able to define a real difference between vendors using
a requirements checkbox evaluation. Another challenge is that a vendor may be an
contract planning function. expert at producing proposals (or hire professional proposal writers) and know how to
address a requirements checklist so that the company looks good, regardless of how
well each requirement is met.
After collating all of the requirements checkboxes and coming down to an even match
between vendors, the decision then becomes one of cost. This is a flawed process that
results in the following questions:
• How can we define our objectives and make the tough trade-offs to determine which
vendor we should choose?
• How can we elicit input from and drive consensus across the evaluation team in an
equitable manner?
• Can we focus the discussion so that we are more efficient in the evaluation process,
while also elevating us from the hidden agendas and inherent biases that others are
bringing to the table?
• How do we tease out the key differences among the vendors to really set them apart,
enabling us to make the “best value” decision?
• How do we know if the vendor’s price is justified by its benefits?
• How can we best defend the decision with senior management and with the ven-
dors?
The members of an organization must work together to assess its relative value—
strengths and weaknesses—across all of the objectives that it’s trying to achieve.
The members need to develop the evaluation criteria as a team, carefully defining the
meaning of each; decide which capabilities are relatively more important than others
in a quantified manner; and finally evaluate the vendors according to strengths and
weaknesses against each of the criteria, collecting all of the relevant comments and
ensuring that the inputs of the evaluation board are captured and aligned in consensus
rather than as checklists of “meets” or “does not meet.”
www.decisionlens.com Decision Lens Proprietary Information, Copyright 2008 Page 2

3. T h e D e c isi o n L en s Ven d or Se lec tion Proce ss
Decision Lens has developed a best-in-class vendor selection process that is used by Fortune 500 companies and agencies across
federal and state governments. This process was developed through an exhaustive 18-month effort to evaluate procurement best
practices across industries, and was then implemented as the core of the “Acquisition Center of Excellence” for the U.S. intelligence
community. It has since been vetted out through hundreds of vendor selections and has proved exceptionally effective at defending
against protests from vendors.
The rigorous process treats the vendors fairly, and each stage is entirely transparent in terms of the evaluators’ priorities and
analysis of vendors’ performances.
Form Source Develop Develop Review Conduct Make Justify &
Selection Statement Proposal & Assess Sensitivity Best-Value Document
Evaluation of Work Evaluation Proposals Analysis Selection Decision
Board (SOW)
Can the evaluation board be grouped one another as possible—mutually
Step 1: Form vendor
into specific functional areas? exclusive and completely exhaustive.
selection evaluation board
Yes, you can divide the evaluation board
The evaluation board for the selection is Step 3: Develop proposal
into Tier 1 and Tier 2 groups. Tier 1
set up with representation from across
groups are at the higher level, evaluating
evaluation
the organization. Often there are two
The list of needs is used to develop spe-
the rollup to the top criteria. The Tier
different evaluation boards created—a
cific selection criteria, which are placed
2 groups are more functionally focused
Technical Assessment evaluation board,
into a hierarchy illustrating all of the
groups that do the feed-in evaluations of
consisting of expert users and opera-
criteria and their relationships to one
the proposals with specific strengths and
tional managers who will evaluate the
another. Criteria may include items such
weaknesses in their areas of expertise.
vendors’ relative technical merits, and a
as technical approach, project man-
Business Assessment evaluation board,
Step 2: Develop state- agement plan, quality of data, system
which will evaluate the vendors’ contrac-
tual terms and conditions, past business
ment of work (SOW) application support and maintenance,
The evaluation board defines a list of corporate quality/experience and ease of
conduct, etc.
needs that are as independent from use. Criteria also may be subdivided into
contributing subcriteria. (See Figure 1.)
Figure 1: Hierarchy of criteria
www.decisionlens.com Decision Lens Proprietary Information, Copyright 2008 Page 3

4. The group defines the relative impor- want the technical evaluation and busi-
tance of the criteria through a set of pair- ness terms evaluation to be untainted by
wise comparisons, asking not only if one cost. Once you have rated each vendor
criterion is more important than another, against the criteria, you are presented
but by how much? with the specific value rating of each
vendor towards your needs.
The Decision Lens tools can be used to
build the criteria, define them and then You can then do a cost analysis by look-
use them to make the decision. The group ing at the relative benefit scores vs. the
creates a value model using all of the costs of the vendors. An independent
quantified criteria. decision is made as to the value/cost Figure 2: Cost vs. value tradeoff
tradeoff, and the final cost analysis If Vendor #1 provides 30% more value
How is cost treated? should not be formulaic; rather, you have and only 20% more cost, you make a
Cost is usually treated as an independent to assess the value of the final configura- cost/benefit tradeoff decision. This is
tion vs. the cost. a much more powerful approach than
variable. You first evaluate the vendors
simply selecting a vendor because it
based on the benefits. You
offers the lowest cost.
How is risk evaluated? criteria. In other words, one criterion of How are the criteria weighted?
Risk should not be included as a sepa- “past performance” may not get a weight The hierarchy is broken down into a
rate criterion. You will be assessing risk that accurately reflects its influence in series of judgments (pair-wise compari-
throughout each criterion in the model. relation to the strengths and weaknesses. sons) at each level. (See Figure 3.)
The risks will be captured in the weak- Instead, we recommend that past per- For example, you are asked the question
nesses that you evaluate for each vendor, formance be evaluated throughout the “Would you give more value to an vendor
as risk for the specific criterion. decision. When participants evaluate the for its technical approach than for its
vendors on their strengths and weak- project management plan?” You would
How is past performance evaluated? nesses under each criterion, the past per- then judge the two criteria being com-
You could develop a criterion for “past formance would be calculated as pared on a scale ranging from 1 (equal
performance,” but past performance part of that. importance) to 9 (extreme difference
should be evaluated across all of the in importance).
Figure 3: Pair-wise comparisons
www.decisionlens.com Decision Lens Proprietary Information, Copyright 2008 Page 4

5. How is the overall value model that will be used to assess the vendors developed?
The result of the pair-wise comparison process is a set of ratio scale priorities showing the value of each of the criterion to the
decision makers. (See Figure 4.)
Decision Lens derives the priorities from the group members and tracks the consistency of their logic and how they applied their
comparisons. This is not forcing them to agree—disagreement is okay—but the process is done explicitly so that all of the judg-
ments across the evaluation board are represented.
Figure 4: Priority graph
What drives the Request for What if there are optional
Proposal (RFP)? When does the RFP requirements? Are those included
process begin? in the upfront hierarchy model?
Don’t send out the RFP until you have If all of the vendors are going to be
developed your selection criteria. The addressing the options in the model, then
criteria will guide the RFP response. You optional requirements should be kept in.
“We needed a decision
should include the criteria and defini- You’ll want to note how important that
process that would tions in the RFP itself, but not option is in your overall consideration
receive buy-in from the weights. as well.
all stakeholders … What if the RFP has already How are rating scales developed?
transmission, been sent? Once you have defined the relative impor-
distribution and IS.” The model can be structured to reflect tance of all of the criteria, you create a
what went into the RFP. Or if you have ratings scale for each criterion. These
Bill Tsolias
specific potential issues not addressed are rulers that will be used to evaluate
Manager, Energy Management Group,
in the RFP, you can develop a model that exactly how well each vendor’s solution
National Grid
exactly matches the RFP and a second addressed the criteria (the strengths and
model to ensure you have captured all of weaknesses of the vendor).
your considerations.
The ratings scales are detailed; an
“excellent” rating has a specific amount
of major strengths, just as “poor” has a
specific amount of major weaknesses.
www.decisionlens.com Decision Lens Proprietary Information, Copyright 2008 Page 5

6. R a t i n g W eigh t De s cri pt ion
Proposal Ratings, Adjectives, and Definitions
Exceptional 1 Offeror’s proposal demonstrates an exceptional understanding of the goals and objectives of the
acquisition. One or more major strengths exist. No major weaknesses exist. Strengths significantly
outweigh the weaknesses. Expected to cause no disruption in schedule, increase in cost, or
degradation in performance. Will require no organizational emphasis and monitoring to
overcome difficulties.
Very Good .8 Offeror’s proposal demonstrates a very good level of understanding of the goals and objectives of
the acquisition. Strengths outbalance weaknesses that exist. Any weaknesses are easily correctable.
Expected to cause minimal disruption of schedule, increase in cost or degradation of performance.
Will require a low level of organizational emphasis and monitoring to overcome difficulties.
Good .5 Offeror’s proposal demonstrates a good level of understanding of the goals and objectives of the
acquisition. There may be strengths or weaknesses or both. Weaknesses are not offset by strengths,
but the weaknesses do not significantly detract from the offeror’s response. Expected to cause
minimal to moderate disruption in schedule, increase in cost, or degradation in performance. Will
require low to medium level of organizational emphasis and monitoring to overcome difficulties.
Marginal .2 Offeror’s proposal demonstrates a marginal level of understanding of the goals and objectives of
the acquisition. Weaknesses have been found that outbalance any strengths that exist. Weaknesses
will usually be difficult to correct. Expected to cause moderate to high disruption in schedule,
increase in cost, or degradation in performance. Will require medium to high organizational
emphasis and monitoring to overcome difficulties.
Unacceptable 0 Offeror’s proposal demonstrates a poor understanding of the goals and objectives of the acquisition.
No major strengths exist, and one or more major weaknesses exist. Weaknesses clearly surpass any
strengths. Weaknesses are expected to be very difficult to correct or are not correctable. This feature
is so poorly understood and demonstrated that it presents an extremely high risk to the success of
the program. Expected to cause significant, serious disruption in schedule, increase in cost, or
degradation in performance. Will require significant or constant, high level of organizational
emphasis and monitoring to overcome difficulties.
(See Figures 5 and 6.)
Figure 5: Example of ratings definitions used in ratings scales
Figure 6: Ratings scales
www.decisionlens.com Decision Lens Proprietary Information, Copyright 2008 Page 6

7. Step 4: Review and
assess proposals
A quick read of the proposals is first
completed. This is designed to help
evaluators calibrate the range of vendor
proposal methods and approaches.
The discussion that results is very
important, and the loss of these com-
ments could be a fatal error. Without
a structured process for gathering
comments and evaluating them against
the criteria, the group will not come
to a logical decision.
With Decision Lens, the comments
are organized right in the software
(see Figure 7) or can be captured in Figure 7: Documenting comments using Decision Lens
an Excel file.
How are a vendor’s strengths and weaknesses evaluated?
Identify the specific strengths and weaknesses of the vendor under each criterion.
Avoid looking at it from a general “meets the requirements” point of view. There are
major strengths and minor strengths, major weaknesses and minor weaknesses. Risks
“Applying Decision Len’s
should be translated into weaknesses.
innovative process for
Each of the participants will write an individual evaluation report for each of the
vendor’s proposals under each of the criteria, listing the major and minor weaknesses
decision making enabled
that he or she has found in the proposal. He or she will evaluate the strengths and our Technical Evaluation
weaknesses (major and minor); include a “Clarification Request” for information in Committees (TEC) to focus
the proposal that is inadequate for evaluation; and identify errors, minor omissions,
misunderstandings and contradictory statements.
on key business and
technical drivers in eval-
What is a Consensus Evaluation Report (CER)? uating vendor proposals.
The evaluation reports written by the participants are then brought together into a
The committees were able
Consensus Evaluation Report (CER). The strengths and weaknesses for each vendor
under each criterion are made explicit in the consensus document and are then used
to quickly and effectively
to rate the vendors. discern major strengths
The CER does not necessarily indicate complete agreement but instead indicates the and weaknesses in ven-
preferred choice of the group. dor proposals and consis-
tently apply their logic in
How are the vendors rated?
Because each rating scale specifically states the strengths and weaknesses required
selecting the best vendor.”
to meet each rating level on the ruler (i.e. “Excellent” or “Poor”), the group is generally Lenetta McCampbell
very aligned in their ratings. In other words, two members of the evaluation board Senior Director,
should not be opposite from one another on the scale if the strengths and weaknesses On Board Systems, Amtrak
are being applied against the scales correctly. (See Figure 8.)
If there is a member of the group who gives radically different ratings than the overall
group, this discrepancy should be addressed. If the group cannot come to agreement
on the ratings, then the difference of opinion must be documented as a comment only
and the group should move on.
www.decisionlens.com Decision Lens Proprietary Information, Copyright 2008 Page 7

8. Figure 8: Ratings
In January of 2006, Harvard Business and political capital. If senior manage-
Step 5: Conduct sensitivity
Review published an entire issue on deci- ment does not agree with the evaluation,
analysis sion making. One of the key criticisms the team should seek the reason for the
Sensitivity analysis is a powerful diag-
was that in most decision-making pro- disagreement (the criteria used, the pri-
nostic tool. It enables the team to test
cesses, there is no means to introduce ority given to the criteria, the rating of
the “what-ifs.” For example, what if the
ad-hoc objections or changes to see how the vendors, etc.). The reason can then
importance of the technical approach is
the decision would be affected. be addressed and evaluated for further
increased? Does that change the priority
refinement. (See Figure 9.)
of the vendors? At what point does one The Decision Lens process enables you to
vendor pass another? introduce changes throughout the model,
providing key insights while saving time
Figure 9: “What-if” sensitivity analysis
www.decisionlens.com Decision Lens Proprietary Information, Copyright 2008 Page 8

9. Step 6: Make best-value selection
Once all of the ratings are complete for each vendor across the criteria, you will have a specific, quantified measure showing the
value of each vendor against the objectives of your organization. (See Figure 10.)
You then may evaluate the relative “value” score against costs, and can determine which vendor best meets your needs. For example,
if you are more cost sensitive, then you may not be willing to make a tradeoff of 30% more benefits from one vendor to another for
just 10% more cost. If cost is less of an issue, you may make this tradeoff.
What if multiple vendors are to be chosen?
Decision Lens optimization capabilities allow you to enter a total cost figure for each vendor. Define your available budget as a
budget pool in the optimizer, and then optimize to get the highest value for cost. The calculation performed in Decision Lens is
to maximize “benefit priority/cost.”
Figure 10: Ratings scoresheet
suited for this process and has been
Step 7: Justify and Conclusion
proven through numerous “best-value”
Document Decision The vendor selection process is indeed
vendor selection decisions to be a true
The power of this process is that each challenging, but when rigor is applied
best-practice approach.
stage is explicitly and rigorously defined through each step of the process with
and quantified. Specific judgments from a sound methodology and the ability to
each evaluation board member are used analyze both qualitative and quantitative
to drive the value of the criteria. Explicit evaluation criteria together, you arrive
strengths and weaknesses and comments at a robust and defensible decision. It is
are used to drive the vendor ratings. The critical that a major vendor selection pro-
group works collaboratively, but consen- cess be able to bring together all of the
sus is not forced at any stage; transpar- stakeholders into a common collabora-
ency and explicit judgment are required. tion that generates buy-in, and that their
As such, any future review or audit judgments, comments and evaluation
has all of the necessary information to points be captured throughout the pro-
debrief senior management, vendors and/ cess as well. Decision Lens is ideally
or your customers.
www.decisionlens.com Decision Lens Proprietary Information, Copyright 2008 Page 9