Short introduction to Delphi method (Futures Studies)

1. DELPHOI
Master’s Degree Programme, FutuS-2 Futures Research Methods
Otavan Opisto, February 9, 2012
Anita Rubin

2. Knowledge interests
Knowledge Interests in Research
Technical knowledge
interest (Delphi)
Practical,
Hermeneutical and/or
interpretative
knowledge
interest(SSM, Delphi)
Intuition (CLA,
Delphi, SSM
Critical
knowledge
Interest (CLA)
Study
problem
Source: P. Kyrö, www.metodix.fi 30 May, 2001
10.11.2014 2

4. Definition
A technique of structuring a communication
process of a group in order to help to
understand and deal with the future
development of a complex problem.
(Linstone & Turoff 1975
• A panel formed by invited experts evaluates the future of a specific
topic /course of development.
• Often replaces and/-or supplements the work of expert groups 
creates material for scenario working (e.g. the variables in a FAR
table).
• Especially useful in exploring large, fundamental issues and aims,
development endeavours, or problems which are socially sensitive
(not to be approached with the help of more ordinary methods).
• Most beneficial as a method when the topic under study is open.
10.11.2014 4

5. Ancient history
The Greek myth tells that in Delphoi, people worshipped Gaia, the Earth
goddess. She had a temple called Pytho and it was guarded by a huge
snake dragon, Python. Apollo, Son of Zeus, killed Goddess Gaia’s dragon
and made himself the ruler of the place. He then changed himself into a
dolphin in order to lure priests for his sailor oracles.
In the course of time, the site of the dolphin, Delphoi, became the
center of the future. There Apollo’s future prophesies were mediated by
Pythias, young and drugged female oracles, whose blurred messages
were translated by priests.
Throughout ancient Europe, Delphoi was famous for its oracles. As
answers to the questions stated by warriors, kings and politicians,
priestess Pythia dictated blurry prophesies in an ecstatic condition,
caused by herbs and psychoactive drugs. Apollo’s priests then
interpreted an formulated the prophesies into poetic form.
10.11.2014 5

6. As a technique, Delphi was first introduced by Olaf Helmer at
the Rand corporation (USA) in the 1950’s and 1960’s.
The Delphoi technique is often also called the Delphi technique
(method).
While Delphi is a specific method to produce new information,
it is also a group of similar techniques to evaluate information
and knowledge already existing  tacit knowledge.
Delphi has also been used in e.g. finding and defining new
technologies and innovations and evaluating the speed of their
expansion (e.g. large studies of technology assessment
involving thousands of panelists in Japan).
6
More history

7. …and still some more history
Originally the aim of a Delphi process was to determine the future
development of some specific topic by the experts in the field. The idea
was to gain a consensus, and this was done by pressing the opinions
through the panels as many times as needed for that.
The method was strongly criticised in the 1970’s from
• leaning on the median in the opinion as the most
competent estimation;
• causing a bandwagon effect on some panelists,
and
• allowing some experts to use anonymity to shape
the general opinion towards his/her direction by
using the process.
 The method was nearly forgotten until the
beginning of the 90’s. Now the method widely in
use  new principles were established.
10.11.2014 7

8. Characteristics
The meaning of a modern Delphi study is to create a mixture of
expert knowledge and tacit knowledge.
The aim is no more to gain a consensus, but to especially dig out
new and fresh viewpoints and ideas and to make the others reflect
on those  many alternative and well-grounded ideas of the
future development.
Today Delphi is used to help decision-making and/or to build
scenarios on a more profound basis.
Six special features:
• anonymity
• iteration of arguments
• communicativeness
• well-founded feedback
• specified expertise
• learning with dialogue

9. Anonymity
The experts in the panel usually do not know
who the other panel members are. They
answer to the questionnaire or survey
program alone.
The advantages of minimising ordinary group
dynamics
 independency from roles
 imposition of authority and strong opinion leaders;
 authenticity of opinions;
 new ideas also from not so well-known/respected experts;
 the shy and the quiet who usually do not bring forth their ideas and
opinions, can safely do so;
 easier for experts to change their opinion (without the fear of ”losing
face” or credibility among colleagues).

10. Iteration of arguments
At first the researcher forms a series of questions or claims and then
gives them to the experts to answer. The experts are asked not only to
cross out the option which best corresponds with their opinion, but also
give reasons why they think the way they think. Then the answers are
analysed and the researcher formulates a new questionnaire where
he/she especially concentrates on those answers which
a) differ from the general way of thinking;
b) include aspects which sound new, fresh, original, or show marks of
new knowledge;
c) open new and different viewpoints or lines for further study;
d) need further clarification;
e) are interesting.

11. …/… Communicativeness
The panelists are given a chance to comment on each
others’ ideas and opinions, question them and give more
information.
 Dialogue between a large number of panelists from
different (but tangenting9 fields of expertise;
 Experts can get new and fresh information;
 Possibility to change one’s mind in the light of new
information;
 Possibility to develop the original topic with the help
of expert discussion.
10.11.2014 11

12. 12

13. Well-grounded feedback
In addition to multiple-choice answers (Likert), the panelists are usually
also asked to give grounds to their opinion – why they pick up just the
alternative they crossed out. The reasons for that might be very
different among the panelists who choose the same alternative.
Feedback consists of analysed results with a special emphasis on
diverging opinions and the reasons/arguments behind those (asked esp.
from those original panelists who gave an out-of-the line answer).
• Both statistical and interpretative data  challenge for the
researcher (e.g. how to prepare valid and relevant theses for the next
round, how to analyse the results for reporting; how to make
scenarios?);
• The expert opinions are not accepted as such  reasoned views (in
the light of latest information) are expected.
• Each opinion can be corrected and changed, based on argumentation
and feedback from the others.

14. Specified expertise
Expertise is the crucial factor in order to make a Delphi process
successful  Osmo Kuusi. The choice is a systematic process to be
carried out e.g. with a table in which the topic under study is opened
from the point of view of (1) experts’ knowledge competences and (2)
interests and background. Therefore the expert should
 be on the top level of his/her field (work, interests, background,
etc.);
 have knowledge and an open mind to other (nearby) fields;
 be able to see interconnections between national/global,
past/present/future courses of development;
 be creative and imaginative, i.e., be able to view the topic from also
unusual and un-orthodox angles;
 be able to see wholes there where others see details;
 be motivated, courageous, open-minded and honest (to accept also
opinions and ideas which differ from those of him-/herself.

15. Expert panel
The ideal size of the panel is 20 to 50 members (no more than 70)
(see later some exceptions).
The panel members can also be found by asking those panelists
who have already been invited to name other experts in the field.
Who are the experts?
 Depends on the knowledge interests of the study:
researchers, decision-makers, entrepreneurs, citizen activists,
teachers of the field topic, key figures in society dealing with
the study problem, etc.
 People who usually are accepted as experts by the other
experts in the field;
 Leading figures in their field of expertise;
 Experts from the tangential fields.

17. 17
A good expert panelist
• Represents an important field of expertise;
• Represents an expert view point which would stay
unnoticed if he/she wouldn’t participate;
• ”Catalyses” the other experts;
• Is interested in the topic and willing to reason;
• Has the guts to take a stance on the topic/theme in
a futures-oriented way.

18. Learning
At its best, a Delphi study is a profound
learning and development process to the
panelists. A panelist’s way of thinking may
change, when (1) the level of knowledge
grows along the Delphi rounds, and (2) his/her
own knowledge and thinking is critically
weighed by the other panelists.
 Deeply rooted basic beliefs get dusted,
aired and if they are good, pollinated;
 The knowledge of the members of a
group creates new levels of knowledge
which may become more than just the
sum of individual panelists’ knowledge
 benefit for the field under this specific
Delphi study.
Jovi Schnell:
Tacit knowledge adrift on the
Myth of Progress

19. 19
The original Delphi
In the beginning, the meaning of a Delphi process was with
the help of experts, to define the future of some specific
phenomenon.
The aim was to gain consensus as the result of the process.
This consensus was reached by moulding and circulating the
opinions in the panel so many times that in the end the
panelists all thought the same way.
(The problems of the original Delphi were discussed earlier.)

20. Characteristics of Delphi today
Today, the information gained from a successful Delphi is a
systemic combination of tacit knowledge and expert knowledge
 visionary knowledge.
The information gained in modern Delphis is needed especially
for the production of such views, ideas and grounds which are
needed for decision-making and planning.
The aim is therefore no more to gain a consensus but especially
bring into light new and fresh viewpoints, knowledge and ideas
and make other experts to react to them  new and well-grounded
20
ideas for possible futures or future states of the topic
under study.

21. • The goal is to reach a consensus among the experts.
• Traditionally large panels.
• The meaning is to gain unanimous and precision forecasts and to utilize
the deviant opinions and ideas in decision-making.
• The consensus can be aimed at only in the last round  benefiting from
the marginal opinions.
• Useful especially when it is necessary to find/create a consensus in a
small unit (group, society, organisation, firm, etc.) in order to build
shared goals (a tool in strategy work).
21
Consensus Delphi (Classical Delphi)

22. Policy Delphi
• The goal is to find/create well-grounded ideas and views of the
future(s);
• Aims at finding the probable and desirable futures/future states;
• Aims also at a set of complementary arguments (e.g. net-based so-called
22
Dialogue Delphi);
• Special attention to divergent views so that as many relevant
thing/idea/factor/course of development/trend etc. can be
considered in decision-making;
• Rather small panels (usually 10-45 panelists);
• A method for social/societal learning;
• A special case: Argumentative Delphi (Argument Delphi) where the
panelists are made to comment on their own claims/opinions  the
goal is the variety of opinions.

23. Other Delphi processes: Trend Delphi
• The starting point is a single trend which is presented as a
23
line/curve.
• The panelists are asked to continue the line/curve and
then give grounds to the course of development they
drew by listing their presumptions and those factors of
uncertainty which might change the line/curve.
• In the end, each panelist is asked to vote about the
different alternative lines/curves presented by the
panelists.

24. Delphi barometers
• Argument Delphi
• The same panel is asked to repeat their answers at certain
intervals (e.g. a year, a quarter year)  motivation.
• The first (and perhaps the second) round are carried out in
order to test the set of questions.
• The idea is to see how opinions on the topics/themes of the
questionnaire develop in time.
• Futures barometer of learning 2030

25. • Online questionnaire
• Does not include sequential rounds;
• Based on qualitative analysis and dialogue;
• In the given time frame, the panelists can visit the
questionnaire, read the comments and discussion by
the other panelists and then react and change their
opinion as often as they wish;
• More normative and communcative than the ordinary
Delphi.
Real-time Delphi

26. • Quantitative
• Large panels (often
26
Survey Delphi
hundreds of respondents)
 computer-based surveys
• The level of expertise is not
very important.
• Often explores the possible
future courses of
technology (e.g. Japan)

27. The phases of a Delphi process 1
A Delphi process can be divided into 11 phases:
1. Definition of the study problem and the goals of the study;
2. Invitation of the research group for planning and execution
27
of the study;
3. Collecting and analysis of necessary background
information;
4. Selection and invitation of the actual expert panel and
sharing of information about the study process 
motivation;
5. Construction of the study questionnaire, testing and
correcting it for the first round …/…

28. The phases of a Delphi process 2
6. Execution of the first study round (either via mail oras an
28
electric questionnaire or face-to-face
interview/questionnaire);
7. Analysis of the results of the first round; )
8. Based on the results of the first round, construction of the
second round questionnaire (argumentative round);
9. Testing, execution and analysis of the results of the 2.
round questionnaire;
10. Third round: construction of the questionnaire based on
the previous rounds, execution and analysis of the answers;
11. (possible extra rounds) and reporting of the results.

29. To be noticed: Experts
The choice of experts can be made carelessly, too roughly, or in
an imprecise way  the validity and reliability of the research
suffer.
To be asked:
• Is this expert really an expert in the exact field/topic needed
in the Delphi?  knowledge competences, involvement,
level of interest, etc.
• As a group, are they representative enough?  too large or
too specified group opinions only mean vague
generalisations
• Is each relevant field represented in an equal way?
• Do they know the development in the near-by fields  if
29
not, some new experts are needed

30. To be noticed: Set of questions/theses
Check the accuracy, adequacy, validity and correctness of
each question.
The less motivated or frustrated experts might leave out
from the next round. Why?
• Unclear, ambiguous or too difficult set of questions, or
• Too wide a questionnaire (takes too much time to
answer)
 use test panelists!

31. To be noticed: Analysis of the rounds
A summary of analysis (between rounds) which is incorrect,
badly written or clings to irrelevant details inevitably affects
the results of the next round:
Coding and analysis incomplete:  in the next round, the
panelists concentrate on irrelevant topics or exciting but
useless little details from the point of view of the actual
research  the validity of the study suffers.
At worst, a badly executed Delphi only strenghtens
conventional, narrow and superficial thinking, and it does not
work as a learning process for the participants.

32. To be noticed: Reporting
The researcher cannot concentrate on the essential  analysis
and thereafter the contents of the report become slanted.
The researher gains results which do not please the customer
 dishonesty in reporting the drawbacks of the study process.
”Secret” study results: lacking possibility for the scientific
community to evaluate them. However, sometimes it is
justifiable not to open all the results to the public (e.g. when
the topic is socially delicate, the panelists might become
endangered, etc.)
The researcher should have a wide understanding of causal
relations; especially the knowledge of social theory is
important.
32

33. To be noticed: Opinions
Figure out how much of the opinions of experts is genuine and
how much is mere repetition of general knowledge/opinions.
Make sure already in the beginning that each expert gives
his/her response as a private individual, not as the
representative of his/her organisation, reference group,
stakeholder, business firm etc., unless this specifically is the
priority.
The further in the future the questions take the panelists, the
less accurate their opinions are.
33

34. 34
No knowledge is more important than the
knowledge of what not yet is, but could be.
Mihail Bahtin