Technique research approach used to gain consensus through a series of round of questionnaire survey

1. Nita Arisanti
Delphi Method for
Community
Assessment

2. Outline
Definition and concept
01
Procedures
02
Consensus measurement
03
Best Practice
04

3. Definition and concept

4. Definition
The Delphi technique is a research approach
used to gain consensus through a series of
rounds of questionnaire surveys, usually
two or three, where information and
results are fed back to panel members
between each round.

5. The Delphi
• The Delphi technique has been used widely in health research within the fields of technology
assessment, education and training, priorities and information and in developing nursing and
clinical practice
• One of the best consensus methods
• Being employed to help enhance effective decision-making in health and social care.
• Delphi as a research methodology has been variously presented as a survey (Wang et al., 2003),
procedure (Rogers and Lopez, 2002), method (Linstone and Turoff, 1975; Crisp et al., 1997) and
technique (Broomfield and Humphries, 2001; Snyder-Halpern, 2002; Sharkey and Sharples, 2001).
• It was first used in technology forecasting studies initiated by the RAND (Research and
Development) Corporation for the American military in 1944

6. When to use Delphi Technique
1. where a problem does not permit the application of precise analytical techniques
but can benefit from subjective judgements on a collective basis;
2. where the relevant specialists are in different fields and occupations and not in
direct communication;
3. where the number of specialists is too large to effectively interact in a face to-face
exchange and too little time and/or funds are available to organize group
meetings; and
4. where ethical or social dilemmas dominate economic or technical ones (Linstone
and Turoff, 1975; Gupta and Clarke, 1996)

7. Characteristics
can be presented either numerically
or graphically
Statistical “group response”
Facilitator decides on the type of
feedback and its provision.
After each Delphi
round, the survey data is statistically
analyzed and re-stated in aggregated
form.
Controlled feedback
The process is coordinated by
a moderator.
Questionnaires are filled in by the
individuals and returned to the
facilitator, who then analyses the
group response.
Anonymity
summarized by the facilitator and
provided as feedback or basic
information for the following round.
This process is usually reiterated
until stability in the responses is
attained
Iteration
03
04
02
01

8. Hierarchical stopping criteria for Delphi studies.

9. Procedures

10. Procedures
Determining initial positions on the
issues
Which are the issues everyone agrees on
and which are the unimportant ones to be
discarded
Which are the ones exhibiting disagreement?
Exploring and obtaining the reasons
for disagreements
What individual underlying assumptions,
views, or facts are being used to support
the panelists’ respective positions?
Exposing the options
Given the issues, what are the
policy options?
Reevaluating
Based on the views of the underlying
evidence and the assessment of its
relevance to each position.
Formulation of the issue
What is the issue that really should be
under consideration?
How should it be stated?
Evaluating the underlying reasons
How does the group view the separate
arguments used to defend various positions,
and how do they compare to one another
on a relative basis?

11. Procedures
• The Delphi is a group facilitation technique that seeks to obtain consensus on the opinions of `experts'
through a series of structured questionnaires (commonly referred to as rounds).
• The questionnaires are completed anonymously by these `experts' (commonly referred to as the
panelists, participants or respondents).
• Qualitative data can be collected through focus groups or interviews and used to inform a quantitative
first round of the Delphi
• The initial questionnaire may also collect qualitative comments, which are fed back to the participants in a
quantitative form through a second questionnaire.
• After statistical analysis regarding group collective opinion, the results from the second questionnaire help
in the formulation of the third quantitative questionnaire. This process is ongoing until consensus is
obtained.

12. Finding a sample
A
Individuals who have knowledge of the topic
being investigated
B
To provide representative information, some
studies have employed over 60 participants
(Alexander & Kroposki 1999)
while others have involved as few as 15
participants (Burns 1998).
The selection of the sample of `experts' involves nonprobability sampling techniques, either purposive sampling or
criterion sampling.
Sampling

13. Data collection
• Typically, Delphi studies include three rounds.
• With regard to the time management between iterations, Delbecq et al. (1975) note
that giving two weeks for Delphi subjects to respond to each round is encouraged.

14. Rounds
Round 1
•element
•element
Round 2
•element
•element
Round 3
•element
•element
• The purpose of the
first round is to
form issues.
• The second round provides the
panelists with feedback from the first
round and presents a questionnaire to
the panelists. The panelists rate the
items on the questionnaire using a
predetermined scale.
• The Delphi moderator analyzed the
data to determine consensus from the
second round.
• Individuals are asked to reevaluate their
opinions in the third round when they
differ significantly from the other
panelists.
• The purpose of the third round is to
provide feedback from the previous
round and to reach a final consensus or
to indicate that consensus cannot be
reached.

15. Data collection

16. Consensus measurement

17. Data Analysis
• Ascertaining the level of collective
opinion often entails the use of
descriptive and inferential statistics.
• For example, round two requires the
data from the ratings of the items to
be analyzed by producing statistical
summaries
The use of participants who have
knowledge and an interest in the topic
may help to increase the
content validity of the Delphi (Goodman
1987)
The use of successive rounds of the
questionnaire helps to increase
the concurrent validity.
The validity of results will be ultimately
affected by the response rates.
• Reliability and Validity
• Subsequent rounds are analyzed to
identify convergence and change of
respondents judgements or opinions.
• Central tendencies (means,
medians and mode) and levels
of dispersion (standard
deviation and the inter-
quartile range).

20. Limitation
A B
C D
Potential of Low Response
Rates
The iteration characteristics of the Delphi
technique can potentially enable
investigators to mold opinions
Potential of Molding
Opinions
Consumption of Large Blocks
of Time
“Some panelists may have much more in-
depth knowledge of certain topics, whereas
other panelists are more knowledgeable
about different topics”.
Potential of Identifying General
Statements vs. Specific Topic Related
Information

21. Best practice

22. Abstract
Background: Chronic respiratory diseases are a major cause of
mortality and morbidity, and represent a high chronic disease burden,
which is expected to rise between now and 2020. Care for chronic
diseases is increasingly located in community settings for reasons of
efficiency and patient preference, though what services should be
offered and where is contested. Our aim was to identify the key
characteristics of a community-based service for chronic respiratory
disease to help inform NHS commissioning decisions.
Methods: We used the Delphi method of consensus development.
We derived components from Wagner’s Chronic Care Model (CCM),
an evidence-based, multi-dimensional framework for improving
chronic illness care. We used the linked Assessment of Chronic Illness
Care to derive standards for each component.
We established a purposeful panel of experts to form the Delphi
group. This was multidisciplinary and included national and
international experts in the field, as well as local health professionals
involved in the delivery of respiratory services. Consensus was
defined in terms of medians and means. Participants were able to
propose new components in round one.
.
Results: Twenty-one experts were invited to participate, and 18
agreed to take part (85.7% response). Sixteen
responded to the first round (88.9%), 14 to the second round
(77.8%) and 13 to the third round (72.2%). The panel
rated twelve of the original fifteen components of the CCM to be
a high priority for community-based respiratory
care model, with varying levels of consensus. Where consensus
was achieved, there was agreement that the
component should be delivered to an advanced standard. Four
additional components were identified, all of which
would be categorized as part of delivery system design.
Conclusions: This consensus development process confirmed the
validity of the CCM as a basis for a community based respiratory
care service and identified a small number of additional
components. Our approach has the potential to be applied to
service redesign for other chronic conditions.

23. Abstract
Objective To identify priorities for the delivery of
community-based Child and Adolescent Mental health
Services (CAMHS).
Design (1) Qualitative methods to gather public and
professional opinions regarding the key principles and
components of effective service delivery. (2) Two-round,
two-panel adapted Delphi study. The Delphi method was
adapted so professionals received additional feedback
about the public panel scores. Descriptive statistics were
computed. Items rated 8–10 on a scale of importance by
≥80% of both panels were identified as shared priorities.
Setting Eastern region of England.
Participants (1) 53 members of the public; 95
professionals from the children’s workforce. (2) Two
panels. Public panel: round 1,n=23; round 2,n=16.
Professional panel: round 1,n=44; round 2,n=33.
.Results: 51 items met the criterion for between group
consensus. Thematic grouping of these items revealed three
key findings: the perceived importance of schools in mental
health promotion and prevention of mental illness; an
emphasis on how specialist mental health services are
delivered rather than what is delivered (ie, specific
treatments/programmes), and the need to monitor and
evaluate service impact against shared outcomes that reflect
well-being and function, in addition to the mere absence of
mental health symptoms or disorders.
Conclusions Areas of consensus represent shared
priorities for service provision in the East of England.
These findings help to operationalize high level plans for
service transformation in line with the goals and needs of
those using and working in the local system and may be
particularly useful for identifying gaps in ongoing
transformation efforts.

24. Abstract
Introduction: The aim of this study was to ascertain what
items stroke survivors and stroke care professionals think are
important when assessing quality of life for stroke survivors
with visual impairment for inclusion in the new patient-
reported outcome measure.
Methods: A reactive Delphi process was used in a three-
round electronic-based survey.
The items presented consisted of 62 items originally sourced
from a systematic review of existing vision-related quality of
life instruments and stroke survivor interviews, reduced and
refined following a ranking exercise and pilot with stroke
survivors with visual impairment. Stakeholders (stroke
survivors/clinicians) were invited to take part in the process.
A consensus definition of ≥70% was decided a priori.
Participants were asked to rank importance on a 9-point
scale and categorize the items by relevance to types of visual
impairment following stroke or not relevant. Analysis of
consensus, stability, and agreement was conducted.
.
Results: In total, 113 participants registered for the Delphi survey
of which 47 (41.6%) completed all three rounds. Response rates
to the three rounds were 78/113 (69.0%),
61/76 (81.3%), and 49/64 (76.6%), respectively. The participants
included orthoptists (45.4%), occupational therapists (44.3%), and
stroke survivors (10.3%). Consensus was reached on 56.5% of
items in the three-round process, all for inclusion. A consensus
was reached for 83.8% in the categorization of items. The
majority (82.6%) of consensus were for relevant to ‘all visual
impairment following stroke’; two items were deemed ‘not
relevant’.
Conclusion: The lack of item reduction achieved by this Delphi
process highlights the need for additional methods of item
reduction in the development of a new PROM for
visual impairment following stroke.

25. Identifying Elements for the Palliative Care Model - A Delphi Study
Nita Arisanti1, Elsa Pudji Setiawati1, Veranita Pandia2, Dany Hilmanto3
1Department of Public Health, Faculty of Medicine, Universitas Padjadjaran
2Department of Psychiatric, Faculty of Medicine, Universitas Padjadjaran
3Department of Child Health, Faculty of Medicine, Universitas Padjadjaran
Jalan Eykman No. 38, Bandung 40161-Indonesia
Abstract
Introduction: Palliative care has been introduced into the
health care system in Indonesia since 1989. In fact, what care
should be offered and performed does not establish. The
study was aimed to design excellence and quality model of
palliative care using Delphi Method.
Methods: A Delphi round was undertaken in three rounds.
Multidisciplinary experts involve in round 1, 2 and 3. The
experts were asked to consider which elements should be
included in the model of palliative care. The elements of care
were derived from previous study resulted 30 elements.
Consensus was defined in terms of median rating, coefficient
of variance (CV) and Kendall’s coefficient of concordance.
The experts were able to propose new elements in first
round. The setting of study was conducted in Bandung
District.
.
Results: The round was completed by 17 experts (GP, nurse,
neurologist, pediatricians) and generated 31 elements. Response
rates to the three rounds were 17 (100%), 17 (100%) and 10
(59%) respectively.
Thirty-one items achieved ≥70% consensus and were included in
the survey.
The absolute value of the difference in CV between Rounds 2 and
3 was less than 0.2, which can be considered to be a minor
difference.
For the final round Kendall’s coefficient was calculated (W=0.527)
and found to be statistically significant (at p<0.05).
Conclusion: The elements have been identified and reach
consensus. The process confirmed the validity of the model as a
basis for the palliative care model. The model has potential
opportunity to be implemented.
Under review

26. Method & Results
Results
• The degree of internal understanding among the
respondents for each element was measured
using the Kendall coefficient (W).
• In the second round, the value was 0.536 and it
was significant.
• In the third round the value was 0.527. This
indicates that the degree of understanding is
quite strong (moderate agreement).

27. References
1. Henderson EJ, Rubin GP. Development of a community-based
model for respiratory care services. BMC Heal Ser.
2012;12(193):1–10.
2. Hasson F, Keeney S, McKenna H. Research guidelines for the
Delphi survey technique. J Adv Nurs. 2000;32(4):1008–15.
3. Tabrizi J-S, Farahbakhsh M, Shahgoli J, Rahbar MR, Naghavi-
Behzad M, Ahadi H-R, et al. Designing excellence and quality
model for training centers of primary health care: a delphi
method study. Iran J Public Health. 2015;44(10):1367–75.
4. von der Gracht HA. Consensus measurement in Delphi studies.
Review and implications for future quality assurance. Technol
Forecast Soc Change. 2012;79(8):1525–36.
5. Hsu C-C, Sandford B. The Delphi technique. Handb Futur Res.
2007;12(10):1–8.
6. Gupta R, Moriates C, Harisson J, Valencia V, Ong M, Clarke R, et
al. Development of a high-value care culture survey: a modified
Delphi process and psychometric evaluation. BMJ Qual Saf.
2016;1–9.
7. Chetty V, Hanass-Hancock J, Myezwa H. Expert consensus on
the rehabilitation framework guiding a model of care for
people living with HIV in a South African Setting. J Assoc
Nurses AIDS Care. 2016;27(1):77–88.
8. Thangaratinam S, Redman CW. The Delphi technique. Obstet
Gynaecol. 2005;7(2):120–5.
9. Barnsley J, Berta W, Cockerill R, MacPhail J, Vayda E.
Identifying performance indicators for family practice:
assessing levels of consensus. Can Fam physician Medecin Fam
Can. 2005;51:700–1.
10. Porcheret M, Grime J, Main C, Dziedzic K. Developing a
model osteoarthritis consultation: a Delphi consensus exercise.
BMC Musculoskelet Disord. 2013;14(1):25.
11. Hepworth LR, Rowe FJ. Using Delphi methodology in the
development of a new patient-reported outcome measure for
stroke survivors with visual impairment. Brain Behav.
2018;8(2):1–9.

28. Thank you
nita.arisanti@unpad.ac.id