Weitere ähnliche Inhalte
Ähnlich wie Foundations of Delphi for Foresight and Group Communication
Ähnlich wie Foundations of Delphi for Foresight and Group Communication (20)
Kürzlich hochgeladen (20)
Foundations of Delphi for Foresight and Group Communication
- 1. © 2011 Turoff 11
Foundations of Delphi
for
Foresight and Group Communications
Murray Turoff
turoff@njit.edu
http://is.njit.edu/turoff
- 2. © 2011 Turoff 2
From: “The People, Yes” by Carl Sandburg
The white man drew a small circle in the sand
and told the red man, “This is what the Indian,
knows,” and drawing a big circle around the
small one, “This is what the white man knows.”
The Indian took the stick and swept an immense
ring around both circles: This is where the
white man and the red man know nothing.”
(A poetic reason for Delphi)
A seer upon perceiving a flood should be
the first to climb a tree – Kahlil Gibran
(Criteria for Study Designers: belief in the results)
- 3. © 2011 Turoff 33
Definition from 1975
Design of a group communication process
structured/tailored around the nature of the
application and the nature of the group
Original paper and pencil rounds
Anyone can change their view
Anonymity or pennames
Scaling theory to promote understanding
Voting to focus discussion
Select “knowledgeable” people
A round took a month – three to five rounds
Respondents 15 to 500
Prediction, policy analysis, conditional forecasts,
planning, significance of contributions, new product
characteristics, etc, etc. (book has many examples)
- 4. © 2011 Turoff 4
Some interesting forecasts
1953 IBM estimates market for only
50 computers in U.S. with IBM 650
1969 GE management claims BASIC is
useless
1975 HP rejects idea of personal
computer before there was one
1979 Tandy expects to sell only 4,000
of the first real portable computer
- 5. © 2011 Turoff 5
Single Subjective Estimates
Personal judgment
As good as data, models and
information to back it up
Expert Opinion
As many examples of wrong ones as
right ones by experts.
Xerox, Video text, Picture Phone,
X.400
How do you know which expert to
use?
- 6. © 2011 Turoff 66
Added definitions and analogs today
A rose by any other name!
Recommender systems
Marketplace Systems
Collaborative Systems
Knowledge structures and systems
Collaborative tagging and Folksonomies
ESPgame.org A beautifully simple Delphi process
Structural Modeling (user created models)
Computer Mediated Communications
The design of inquiry systems by philosophy
Scientific: Leibnitz, Locke, Kant, Hegel,
Singer – C. West Churchman book
Non Scientific! Heidigger (Negotiated reality,
management and marketing)
Virtual Organizations, teams, groups
- 7. © 2011 Turoff 7
Online and Real Time Delphi or Asynchronous
Collaborative Systems
Today this is the future of the Method and its
Evolution
Many examples already exist and some
experimental evaluation efforts have been
done
In many of these Anonymity, Pen Names or
Signatures by authors (when they choose) or
by the sponsors is a dynamic option
As usual in the computer field technological
options are expanding faster than they can
be evaluated except by looking at the results.
Understanding the successful paper
approachs is still a good guide to how such
systems work
- 8. © 2011 Turoff 8
Objectives of a Delphi Process
Promote exploration of a complex problem
Gathering ideas and information from the
literature and the respondents
Provide an understanding for the group of the
viewpoints of individuals, subgroups and the
whole group.
Use of voting and scaling theory
Expose and Explore disagreements
Encourage focused discussion
Encourage participants to focus on what they
know well
Allow participants to change their minds
- 9. © 2011 Turoff 9
Paper and pencil vs. On line Dynamic Delphis
Paper and Pencil
Specific round structures usually 2 to 5 for paper
and pencil exercises.
Weeks between rounds
Much use of paper, scissors, scotch tape, and Xerox
to eliminate duplications
On Line
Any individual may deal with the part of the
problem they prefer to work with at the moment
Computer tracks all communications and knows
what any participant has or has not seen or
responded to
For Both
Use of a knowledge structure to capture
information and organize it to avoid information
overload
- 10. © 2011 Turoff 1010
Delphi does not make decisions
Delphi should never make a decision, it should be
used to analyze a potential decision or action and
provide that to a single individual or role that is
accountable for the decision to be made.
The decision maker for real time decision should
always be one of the participants
In Emergency management many different roles
make decisions that can interfere with one another
and an Emergency System must track the decision
process to insure everyone has access to the most
current relevant information
This type of system alerts people to conflicts and
those responsible for oversight. The problem solving
Delphi (later) is useful in that type of situation.
It is unpredictable what Roles (active 24 hours 7 days
a week) are needed at a given time for a given problem
but it is always a group that is needed.
The person in the role changes dynamically
- 11. © 2011 Turoff 1111
Characteristics of Delphi I
A content structure appropriate to the problem being
addressed
Automatic organization of content contributions
Dynamic knowledge base creation
Anonymity of voting and degrees of anonymity for
contributions
Forced, voluntary, and pen names
Some groups are told who other members are to
convince them of quality of group (peers)
Anyone can change their original vote
Primary goal of votes to expose different views and
generate discussion of causes for that
Ambiguities, uncertainties, misunderstandings, and
disagreements
Need to eliminate ambiguities before other issues can
be resolved
- 12. © 2011 Turoff 1212
Characteristics of Delphi II
Classifying participants so that votes can be seen
viewed by backgrounds of voters very important for
large studies with very heterogeneous groups
Use of voting and comments is to encourage
participants to change their views based upon relevant
contributions of others.
This is a different objective than surveys and
associated assumption about only measuring a
fixed state of mind (psychological characteristics)
that do not change for an adult
Not all survey guidelines and associated statistics
apply to a process intended to deal with complex
problems and causing people to rethink their
viewpoint
Pennames extremely useful for allowing continuity of
thought across a set of comments and for execution of
role playing learning games or story telling
For example: comment on the following policy
resolutions as the company auditor would
- 13. © 2011 Turoff 13
Motivations for participant
Want to exchange useful information with other
professionals about a complex problem
It is a peer group and or people you don’t have
access to
The facilitators have already gathered all the obvious
material to structure and seed the process
You are not going to educate them and they will
understand what you have to contribute
The problem is important, part of the job, and/or does
not have an obvious solution
It can be a process to test ideas
It is a consulting task that is paid for
The sponsor will take the results seriously
The results will be a useful document for a larger
audience
- 14. © 2011 Turoff 1414
Ultimate Delphi Goal: Collective Intelligence
Group problem solving result is better than
any single member could have done before
the discussion process
Possible with even paper and pencil
Group can solve problems as fast as any
single member considering the same relevant
materials (asynchronous operation)
Possible with computers and Structured
Computer Mediated Communications
- 15. © 2011 Turoff 1515
Delphi Examples:
These have been done very often and lend
themselves to online operation
Trend Delphi: produces a forecast of a trend along
with the mental model of the group making the
extrapolation of the trend curve into the future
Problem Solving Delphi: Collects solutions to the
problem which are rescaled to a group interval scale
based upon individuals ranking or paired comparisons.
Use voting to focus discussion on items that need it.
Policy Delphi: seeks policy resolutions and the
strongest pro and con evidence or arguments to
support each policy resolution
Cross Impact Modeling: Collaborative building of a
model of the future possible outcomes of a set of
unique events.
- 16. © 2011 Turoff 1616
Creativity Experiment
Open ended idea generation
Distribution of total Raw, Unique, and Rare Ideas
Online Discussion System alone and with Delphi
Voting imposed x group size.
Problem: product ideas for a pill sized data responder,
ranked list report.
Notes: Unique ideas (duplications removed), rare ideas
(occurred in 3 or less groups); Statistically significant
on per group and per person that Delphi leads to more
improvisation or creativity.
Cho, H. (2004) The effect of Delphi structure on small
and medium-sized asynchronous groups, Ph. D.
Dissertation, January 2004, New Jersey Institute of
Technology, Information Systems Department.
http://www.library.njit.edu/etd/index.cfm
- 17. © 2011 Turoff 1717
Number of Idea Types
Structure x Group Size
44 groups (11 groups in
each condition)
Small
5-7
persons
Medium
11-14
persons
Delphi:
Raw ideas (duplicates) 157 247
Unique ideas 67 111
Rare ideas (g<=3)) 40 57
Unstructured:
Raw ideas (duplicates) 108 192
Unique ideas 52 86
Rare ideas (g<=3) 19 48
Results for total ideas and ideas per person are both statistically
significant. No reduction in performance for individuals in
medium group. (open ended brain storming problem)
- 18. © 2011 Turoff 1818
Easy Online Delphi Used for Experiment
Bulletin Board System imposed structure by
monitor
Good discussion thread structure
Root items for main ideas
Replies for discussions of each idea
Allowed anonymous or real signatures
Web survey system where monitor collected
and put the ideas for a vote and provided
summaries online
Once a week for three weeks
Non structured condition
Same system but no imposed structure by
monitor and no surveys done
- 19. © 2011 Turoff 19
Trend Forecasting Delphi Round One
Provide one or more historical curves of
interest (measures of success or failure,
sales, enrollments, profit)
Ask for:
Future projection of a historical curve
3-5 Assumptions used to draw the curve
Uncertainties that would cause you to
change the curve you drew
Assumptions & uncertainties:
Modifications, competitors actions,
government policies, economic
conditions, investments, etc.
- 20. © 2011 Turoff 20
Forecasting Delphi Round Two
Turn ALL into potential assumptions
One persons assumption is another’s
uncertainty
Ask for vote on validity:
Certain, Likely, Maybe, Unlikely,
False
Show limits of 50 % spread of drawn
projections
Spread increases with time into the
future
- 21. © 2011 Turoff 21
1
CERTAIN (Average of 1 to 1.5)
•Low risk of being wrong; Decision based upon this will not be wrong
because of this 'fact’; Most inferences drawn from this will be true.
2
RELIABLE (Average of 1.6 to 2.5)
•Some risk of being wrong; Willingness to make a decision based upon
this; Assuming this to be true but recognizing some chance of error;
Some incorrect inferences can be drawn.
3
NOT DETERMINABLE: (at this time) (Average of 2.6 to 3.5)
•The information or knowledge to evaluate the, validity of this assertion
is not available to anyone --expert or decision maker.
4
RISKY (Average of 3.6 to 4.5)
•Substantial risk of being wrong; Not willing to make a decision based
upon this alone; Many incorrect inferences can be drawn; The converse,
if it exists, is possibly RELIABLE.
- 22. © 2011 Turoff 22
5
UNRELIABLE (Average of 4.6 to 5)
•Great risk of being wrong; Worthless as a decision basis; The
converse, if it exists, is possibly CERTAIN.
6
NOT PERTINENT/RELEVANT (Used to eliminate some
assumptions from exercise) Even if the assertion is CERTAIN or
UNRELIABLE it has no significance for the basic- issue; It cannot
affect the variable under question an observable amount.
blank
NO JUDGMENT
•No knowledge to judge this item, but the appropriate individual
(expert, decision maker) should he able to provide an evaluation I
would respect.
Validity and/or Confidence Scale
- 23. © 2011 Turoff 23
Forecasting Delphi Round Three
Show ordered assumptions from
completely true to completely false
Ask for modifications to original
projection
Focus on most important “Maybe”
range
Ask for significance of impact of assumptions
controllable or those not controllable
Determine actions organization can take to
force outcome of controllable events
Determine measures/signals of other external
assumptions becoming true
- 24. © 2011 Turoff 24
Forecasting Delphi Round Four and “maybe”
Five
May not be same group, usually more
decision makers in this part of the process
Determine desirability and feasibility of
actions
Determine usability of measures or ways to
bring about better intelligence
Round five for significant disagreements
Next phase
Management group to do allocation of
resources to influence or measure things
that can change forecast
- 25. © 2011 Turoff 25
You have to have a design for all of the process
Request
Curve Projection
Assumptions and Uncertainties
Turned into Potential Assumptions
Requested Vote on Validity
Provide List of Assumptions from True to False
Request new curve estimate
Extract Controllable events and Uncontrollable in the
Not Determinable range
Examine these events for influencing and
measuring
See Steel and Ferroalloy Delphi article from The
Delphi Method Book for examples
- 26. © 2011 Turoff 26
Results of Trend Delphi
A numeric projection and spread which
might not differ from a regression analysis
A “collaborative knowledge model” of the
groups reasons (assumptions) for the
projection
A collaborative knowledge model of all the
factors that could change the forecast
Three rounds
A plan for:
Trying to influence the outcome
Monitoring for surprise changes from
external sources like competitor actions
Two more rounds
- 27. © 2011 Turoff 27
Modeling Delphi for the Steel Industry
Flow diagram by three experts
45 flow of material legs between different
processes in the industry
Only 15 legs reported in yearly data
40 other experts (planners) asked to fill in missing
data for prior year
Not asked to do anything else
25 decided to modify the model because they did
not agree with it being the “best” model for them
Modeling Delphis need more exploration
Very risky to listen to only one text book, one expert,
or one professor!!!!
Review the literature always necessary before
design of a Delphi
- 28. © 2011 Turoff 28
Policy Delphi I
Major Policy Issue
List obvious resolutions
Vote on Desirability and Feasibility
Allow more proposed
Show results and disagreements
Ask for pro, con, and neutral arguments on
resolutions
Vote on new resolutions
Vote on arguments for Importance and Validity
Continue rounds as long as vote changes
are made and new resolutions emerge
Agreement on underlying comments often lead
to a synthesis or a better resolution
Not necessary
- 29. © 2011 Turoff 29
Policy Delphi Scales
Forcing Specificity of Responses
Policy Resolutions
Very Desirable, Desirable, Undesirable,
Very Undesirable
Definitely Feasible, Possibly Feasible,
Possibly Unfeasible, Definitely Unfeasible
Comments on Resolutions: pro, con, neutral
Very Important, Important, Slightly
Important, Unimportant
Certain, Reliable, Risky, Unreliable
No Judgment always a choice on anything
Easy to make dynamic
- 30. © 2011 Turoff 3030
Why Delphi: Human Considerations
Too many participants needed for face-to-face
Three to five in any given type of expertise and/or
experiences to ensure all rationales are exposed
Early experiments by Dalkey on rationales for specific
subjective
With N types needed for a complex problem this is 3xN to 5xN
participants required
Different backgrounds that require elimination of ambiguity
and misinterpretations as well as translations of concepts
Severe disagreements to be mediated
Freedom of expression and improvisation of ideas without
loss of face
One out of ten ideas valuable
High status participants most concerned about this
Equal Participation allowed
Minimize group process losses
An idea can be brought up at anytime
- 31. © 2011 Turoff 3131
Critical Success Factors
Composition and quality of the participants
Results only as good as the people involved
Compensation for Effort
Communication with a peer group
Letting participants nominate others helps (snowball)
Will learn from others in effort
Results will serve useful and important purpose
Consulting pay if above not strong enough
What is obvious already included, participants used
for what is not obvious (no blank sheets of paper)
A morphological structure to automatically organize
input material
Important functions and objectives: exploration,
understanding one another, exposure of issues and
uncertainties, examination of disagreements, and
generation of agreements when possible
- 32. © 2011 Turoff 3232
Why Delphi: operational challenges
Subjective judgments required
Models to support consistency of judgments by
individuals and group
Building collective models (e.g. cross impact modeling,
structural modeling)
Producing collective Group views (e.g. Arrow’s paradox and
scaling methods)
Individual human biases (~10)
Group process losses and gains (~20)
Avoiding consensus (Asch & Hawthorne effects) pressure
Participant effort and shadow time
Asynchronous flexibility to use any time
Language or cultural difficulties
Virtual teams
Detecting differences by backgrounds for feedback
Multitasking, Cognitive Limitations, Information Overload
- 33. © 2011 Turoff 3333
Delphi technology and methods
Exploring
Capturing individual knowledge
Process of design, problem solving, derivation,
knowledge structures
Animation methods
Promoting Understanding
Forming a group synthesis
Scaling methods
Feedback to group
Social judgment & voting theory
Finalizing Viewpoints (no more changes)
Evaluation by the group
Collaborative processes
- 34. © 2011 Turoff 34
Preventive Security Measures in IS
Management of IS Course (Graduate Students 50% working)
Online system over three weeks
Preventive Measures 72 Total Words (approx.) 25,000
Comments on 433 Average comment size
(words)
480
Pro comments 200 Contributors 20
Con comments 104 Voters 27
Neutral Comments 129 Contributions/person 23
Modifications 25 Contributions/day 25
Total of all items 530 Comments/Measure 6
- 36. © 2011 Turoff 3636
Rank Scale Disaster Damage Dimension
20 20.00 Causalities and fatalities
19
18 18.00 Utilities Impact
17
16 16.00
15.90
Potential to spread
Ability of local response adequacy
15 15.43
15.40
15.40
15.38
Loss of command and control
Infrastructure damage
Resources for aid or containment
Time needed to respond
14 14.82 Duration of disaster
13 13.09 Public reaction
12 12.95 Geographic impact
11
10 10.07 Time to return to normal
9
8 8.61 Chance of imminent reoccurrence
7
6
5
4 4.70 Financial Loss
3
2
1
0 0.01 Financial recovery costs
Feedback: a Thurstone Scale for
Relative
Importance of Measures of
Disaster Impact
Graduate Student class Delphi in an
Emergency Management class with
about 30% with real experience in
the area.
- 37. © 2011 Turoff 3737
Online Delphi’s Today
Dynamic: entering main voting items, voting, making vote changes,
and pro, con, neutral discussion comments, can all go on at once.
Each individual can focus on what they want to
System notifies user of changes
Termination of vote changes is a sign of finalization of the
results
People not confident about a given item can always not vote or
defer their vote to later
People can propose rewording (Roberts Rules of Order)
What took three months can happen in three weeks for complex
problems and large groups
Straight forward problems (e.g. emergency allocation priorities) can
be done in real time by dispersed virtual groups of 5-20
professionals.
Typically voting anonymous, comments signed
Voting really serves to focus discussion on differences of view
Used as a collaborative learning tool in online class discussions
System designed as a Social Decision Support System or Dynamic
Delphi
- 38. © 2011 Turoff 3838
Approaches to the Future
Goals affect what questions to ask
Automation
Reduce people time and effort
Eliminate jobs
Productivity
Increase quantity
Increase quality
Opportunity (preferred for Foresight
studies)
Do new things
Do things differently & Better
- 39. © 2011 Turoff 3939
Approaches to the Future
Dimensions of Group Communications
Information Exchange
Pooling of current data and information
Cooperation
Informing of plans and actions/efforts
Coordination
Joint planning of actions/efforts
Collaboration
Working together on the same
actions/effort
Too often above words are used as if they
were all the same
- 40. © 2011 Turoff 40
Possible Delphi Use in Planning
(Continuous Organizational Collaborative Process)
Problem Solving Delphi
Pick key variables for analysis
Trend Analysis Delphi
Generate Key Potential Assumptions/Events
Problem Solving Delphi
Evaluate Important/Significant events
Cross Impact and ISM
Develop Integrated Model and Simplify Complexity
Policy Delphi
Expose and Investigate disagreements on Decision
and Policy options
Start at the top again for unresolved situations
- 41. © 2011 Turoff 41
End of First Talk
Questions or Comments
Carefully
looking at
the future!
- 42. © 2011 Turoff 4242
Appendix Slides:
Examples of general types of Delphi
and
Dynamic Delphis
These slides represent more detail on
the specific structures mentioned and
also provide a list of references for
those that want to dig deeper.
- 43. © 2011 Turoff 4343
Examples of General Delphi Designs
and
Dynamic Delphis
These have all been done with paper a number of
times
Some have been done partially or totally online or in
combination
They could all be done online in a mode where anyone
can work on any part at anytime
Some of these still require a human monitor to
perform certain functions and some can be fully
automatic
Users can:
1. Add new items or content
2. vote on any item, change their vote, and/or
choose not to vote, and/or vote later after
discussion
3. Suggest alternative wordings to existing item or
add a new version of an existing item.
- 44. © 2011 Turoff 4444
System operations
Allow lists to which options, solutions, goals, etc can
be added
Allow voting scales for such lists
Allow comments on list items
Provide each user with new items of any type and also
any whole discussion they want to see
Provide vote summaries to indicated the relative
status of voted items and highlight differences
Determine any different vote patterns by participants
with different expertise and/or experience, and/or
backgrounds
Show for any item
Number voting
Number of vote changes
Number who might yet vote
Number of discussion entries about an item
- 45. © 2011 Turoff 4545
Integrated Collaborative Planning I
1. Problem solving Delphi Process: Determine and
consider the variables that measure the future for any
organizational objective and use the historical data to
allow a projection of both a desirable and undesirable
future that might define the range of uncertainty.
2. Trend Analysis Delphi Process: Use a trend analysis
collaborative process to determine the events that will
influence both the desirable and undesirable
outcomes. Events are any type of policies, actions,
decisions, resource allocations etc that can influence
the future of the key variables.
3. Problem Solving Delphi Process: Evaluate these options
to reach agreements on which are the most important
for influencing the future. Make sure to use a
balanced set that reflects a rational assessment of
what makes the difference between a desirable and an
undesirable future.
- 46. © 2011 Turoff 46
Integrated Collaborative Planning II
4. Develop a cross impact model for these unique events
and use Interpretive Structural Modeling to take the
subjective judgments of those involved and determine
out of what is potentially a very large event set the
most consistent micro scenarios to reduce this large
event set to a more manageable subset of mini
scenarios.
5. Use these mini scenarios and how they influence one
another to actually build a working model that allows
individuals test the significance of changes in
influencing (general resource allocation) the outcomes
that these scenarios represent. This model can be
treated as a game that professionals can use to test
different alternative decisions and resource
allocations.
6. Use the Policy Delphi to actually resolve
disagreements or judgment uncertainties about
possible decision and policy options. This might
cause restarting the process for some aspect of the
planning problem
- 47. © 2011 Turoff 4747
Trend Delphi
System Functions Participants Responses Response System Actions
Present a historical trend to be
extrapolated by the
participants
Draw a future curve or
redraw a new one when
a change has occurred
in viewpoint.
Present summary of 50%
median and 0%, 25%,
75% and 100%
boundaries
Request assumptions and
uncertainties used to make
above estimate
Turn all these into potential
assumptions
Vote on validity scale for
each potential
assumption.
Scale is from completely
true to completely false.
Reorder assumptions from
true to false.
Focus on middle range
(maybe) and ask which
can be influenced or
measured for occurrence
Assume these can reduce the
future uncertainty in the
curve
Ask for a redrawing of curve
extrapolation based upon
assumption list for each
trend curve in the study
Supply suggestions on how
to influence or measure
the maybe assumptions
causing significant
uncertainty in the
projected curve.
Summarize important
findings at any time:
Trend, true and false
assumptions, assumptions
that cause uncertainty,
and their potential
actions, and
measurements
- 48. © 2011 Turoff 4848
Problem Solving Delphi
System Functions Participants Responses Response System Actions
State the problem and request
solution options
Provide options to solve
the problem
Present options in order of
occurrence
Request paired comparisons to
measure individual
preferences for options
Make comparisons for
option pairs that a
participant feels
confident about
judging at any time.
Use Thurstone's law of
comparative judgment
(using incomplete
information) to derive a
single group interval
scale.
Calculate uncertainty due to
those who have not yet
voted with same type of
scale.
Show interval scale.
This indicates disagreements
when two or more items are
close together. This also
shows clustering. Ask for
comments about items
where people disagree with
current position of an item.
Make comments about
items you want to see
others change their
votes about.
Present discussions about
items for review. As
more people vote or
change votes scales will
reflect decreasing
uncertainty and often
more separation between
options.
- 49. © 2011 Turoff 4949
Policy Delphi
System Functions Participants
Responses
Response System
Actions
State a policy issue to be
examined. Ask for specific
policy solutions
Add resolution options or
specific policies
Request vote for Desirability
and Feasibility scales of
each solution
Plot two dimensional
distribution of policy
resolutions
Exploring desirable but
infeasible solutions often
important
Request comments
especially about those
showing disagreement
Request comments about
policy resolutions.
Indicate if comment is
pro, con, or neutral.
Request vote on comments for
importance and validity
It might be considered
important because others
believe it to be true
A person may think a
comment is important
because others think it
is valid.
Do same two dimensional
plots and summarize
discussions
- 50. © 2011 Turoff 5050
Cross Impact Modeling I
System Functions Participants Responses Response System Actions
Use problem solving Delphi to
produce a set of future unique
events focused on a given
situation
Evaluate those events for
their relative
importance to the
future objective
guiding the choice of
events
Place the final most important
events into a cross impact
model
Ask each individual to answer:
What are the probabilities of each
event occurring in some future
time frame?
Tell them for each event that they
should assume it will or will
not occur and ask them to
express any changes in the
probabilities of the other
events due to that certain
knowledge about the future.
Show them the expected
outcome of their
model which will have
differences from their
predictions. Allow
them to vary initial
probabilities to see
how the future
changes.
Allow them to go back
and modify some of
their estimates
Create the cross impact model
using the approach by Turoff
(logistic, Fermi Dirac
equations). This provides a
scale changes from nonlinear
probabilities (0 to 1) to a
linear influence factor
between each pair of events
(plus to minus infinity).
When participants are satisfied
with their individual model
utilize the internal linear
influencing factors (Cij) to
create a group model.
- 51. © 2011 Turoff 5151
Cross Impact Modeling II
Creating Scenarios
System Functions Participant Responses Response System
Actions
Analyze the internal parameters
to show people which of the
relationships between which
events show the most
disagreement among the
group.
Ask for comments on these
combinations form those
who have inconsistent or
extreme views.
Show these comments and
others and allow those
who wish to change
some of their original
estimations.
Create a model of
interacting scenarios by
voting on where to stop
the integration of the
events in process that
can turn all the events
into one scenario
When no more changes are
being made produce.
Use Interpretive
Structural modeling to
generate a set of macro
scenarios collecting
individual events that
are tightly coupled into
a set of scenarios that
interact.
Requires human monitor to
know when to trigger
the scenario creation
part
- 52. © 2011 Turoff 5252
1972 Paper Example
The Delphi Method Book
Event
number
Description
1
The U.S. gets in a trade war with one or more of its major trading partners (Japan, Canada, western European
countries).
2
Comprehensive Tax Revision S Enacted With Most Present exemptions And Exclusions Removed, But With
Rates Lowered.
3 Rigorous anti-pollution standards are adopted and strictly enforced for both air and water.
4 The U.S. averages at least 4 percent per year growth rate of real GNP for the time frame.
5 Defence spending declines steadily as a percent of the federal government's administrative budget.
6
The U.S. Experiences At Least One Major Recession (GNP Decline is greater than 5 Percent for a duration greater
than 2 quarters) during the ten year period.
7
A federal income maintenance system (e.g., negative income tax) replaces essentially all current state and local
welfare programs.
8 The oil import quota system is phased out and domestic oil prices allowed to fall to the world price.
9 The U.S. agricultural price support system is dismantled.
10
A federal-state and local revenue-sharing program is adopted which allocates at least 5 percent of federal revenues
to state and local governments.
Table 1 Events
- 53. © 2011 Turoff 5353
Use of Interpretative Structural Modeling
with Cross Impact Analysis
Each cycle is a mini scenario or possible new single event
- 54. © 2011 Turoff 54© 2009 Turoff 54
Cross Impact Additions
Decide which events you can replace
with micro scenarios to reduce event
complexity.
Add objective and goals as event sinks
Add source events as investments
Develop offense and defense events to
create a game between two players or
two teams
Can model natural and man made
disasters for exploration and training
- 55. © 2011 Turoff 5555
The following slides contain references
for those that want to learn more
- 56. © 2011 Turoff 5656
Papers on my website 1
Complete copy of original reference book on the Delphi Method
The Delphi Method: Techniques and Applications by Harold
Linstone and Murray Turoff, 1975. (http://is.njit.edu/turoff)
Turoff, M., Hiltz, S. R., Li, Z., Wang, Y., Cho, H., Yao, X.,
(2004),Online Collaborative Learning Enhancement through the
Delphi Method, Proceedings of the OZCHI 2004 Conference,
November 22-24, University of Wollongong, Australia
Murray Turoff, Starr Roxanne Hiltz, Michael Bieber, Ajaz Rana
(1998), Collaborative Discourse Structures in Computer Mediated
Group Communications, 1998
Turoff, Murray, and Starr Roxanne Hiltz (1996), Computer Based
Delphi Processes, a version will appear as an INVITED BOOK
CHAPTER for Michael Adler and Erio Ziglio, editors., Gazing
Into the Oracle: The Delphi Method and Its Application to Social
Policy and Public Health, London, Kingsley Publishers (in press).
- 57. © 2011 Turoff 57
Papers on my website 2
Turoff, Murray (1997) Alternative Futures for Distance Learning:
The Force and the Darkside, The material in this paper was
utilized for an Invited Keynote Presentation at the UNESCO /
OPEN UNIVERSITY International Colloquium, April 27-29:
Virtual Learning Environments and the Role of the Teacher, Open
University, Milton Keynes. It also forms the basis of a planned talk
at the Third International ALN (Asynchronous Learning
Networks) meeting in NY City, October, 1997. (Presentation
Overheads for ALN conference, October, 1997, NYU)
Turoff, Murray (1995), Software Design and the Future of the
Virtual Classroom, Journal of Information Technology for
Teacher Education, Vol 4, No. 2, 1995
Turoff, Murray (1999), An End to Student Segregation: No More
Separation Between Distance Learning and Regular Courses. A
summary of the invited plenary for the Telelearning 99 meeting in
Montreal, Canada, November, 1999. (Also: ppt presentation used
in talk.)
- 58. © 2011 Turoff 5858
Other Papers 1
Turoff, M., Hiltz, S.R.: The Future of Professional Communities of
Practice. In: Weinhardt, C., Luckner, S., Stößer, J. (eds.) WeB
2008. LNBIP, vol. 22, pp. 144-158. Springer-Verlag, Berlin
Heidelberg (2009)
Xiang, Y, Turoff, M., and Chumer, M. Designing a group support
System to Review and Practice Emergency Plans in Virtual Teams,
Proceedings of the 6th International ISCRAM Conference,
Gothenburg, Sweden, May 2009 (http://iscram.org)
White, Connie, Murray Turoff, Bartel Van de Walle, A Dynamic
Delphi Process Utilizing a Modified Thurstone Scaling Method:
Collaborative Judgment in Emergency Response, Proceedings of
ISCRAM 2007, 4th International Conference on Information
Systems for Crisis Response and Management, Delft, the
Netherlands, May 13-16, Brussels University Press
Plotnick, Linda, Elizabeth Avey Gomez, Connie White, Furthering
Development of a Unified Emergency Scale Using Thurstone's Law
of comparative Judgment: A progress Report, Proceedings of
ISCRAM 2007, 4th International Conference on Information
Systems for Crisis Response and Management, Delft, the
Netherlands, May 13-16, Brussels University Press
- 59. © 2011 Turoff 59
Other papers 2
Turoff, Murray, Starr Roxanne Hiltz, Xiang Yao, Zheng Li,
Yuanqiong Wang, and Hee-Kyung Cho, Online Collaborative
Learning Enhancement Through the Delphi Method, Turkish
Online Journal of Distance Education-TOJDE April 2006 ISSN
1302-6488 Volume: 7 Number: 2 Article: 6, Publisher: Anadolu
University, Eskisehir, Turkey,
http://tojde.anadolu.edu.tr/index.htm
Hee-Kyung Cho, Murray Turoff, and Starr Roxanne Hiltz, The
Impact of Delphi Communication on Small and Medium Sized
Asynchronous Groups: Preliminary Results, HICSS 36, January
2003, IEEE Computer Society Press.
Turoff, M., Hiltz, S. R., Li, Z., Wang, Y., Cho, H. "The Delphi
Process as a Collaborative Learning Method." In (edited by J. C.
Moore) Elements of Quality Online Education: Into the
Mainstream: Wisdom from the Sloan Consortium, 121-134.
Needham, MA: Sloan-C, September 2004
Banuls, V., and Turoff, M., Scenario Construction via Cross-
Impact, Draft under review 2009.
- 60. © 2011 Turoff 6060
Other Papers 3
Cho, H.K. & Turoff, M., “Delphi Structure and Group
Size in Asynchronous Computer-Mediated
Communications,” Proceedings of the Americas
Conference on Information Systems, Tampa, August
2003.
Wang, Y., Li, Z., Turoff, M. and Hiltz, S.R. (2003).
Using a social decision support system toolkit to
evaluate achieved course objectives. Proceedings of the
Americas Conference on Information Systems, Tampa,
August. (Nominated as a “best paper.”)
Turoff, Murray and S. R. Hiltz, (1995), Computer
Based Delphi Processes, in Michael Adler and Erio
Ziglio, editors., Gazing Into the Oracle: The Delphi
Method and Its Application to Social Policy and Public
Health, London, Kingsley Publishers, pp. 56-88.
- 61. © 2011 Turoff 61
Other papers 4
Worrell, W., Hiltz, S. R., Turoff, M. and Fjermestad, J. (1995) An
experiment in collaborative learning using a game and a computer-
mediated conference in accounting games. Proceedings of the 28th
Annual Hawaii International Conference on System Sciences, Vol.
IV, pp. 63-71. Los Alamitos, CA: IEEE Computer Society Press,
1995.
Hsu, Enrico Y. P., Hiltz, S. R., and Turoff, Murray (1992).
Computer-Mediated Conferencing System as Applied to a Business
Curriculum: A Research Update. In V. S. Jacob and H. Pirkul,
eds., The Impact of Information Technology on Business Schools:
Research, Teaching and Administration, Proceedings of the 20th
Annual North American Conference of the International Business
School Computer Users Group, pp. 214- 227. Awarded "Best
Paper- Teaching.“
Hiltz, S.R. and Turoff, M., The Network Nation: Human
Communication via Computer, 1978, revised edition reprinted
1993 by MIT Press