Professor John N Lavis, Director of the McMaster Health Forum at McMaster University in Canada, recently addressed a CIPHER forum to share his experience in making research useful for health decision makers. CIPHER, the Centre for Informing Policy in Health with Evidence from Research, is an Australian collaborative research centre managed by the Sax Institute, that is investigating the tools, skills and systems that might contribute to an increased use of research evidence in policy. For more information visit www.saxinstitute.org.au.

1. 1
International Perspectives
Making Research Work for Decision Makers: Innovations and Priorities for Action
Sydney, Australia
John N. Lavis, MD, PhD
Professor and Director, McMaster Health Forum
McMaster University
Adjunct Professor of Global Health
Harvard School of Public Health

2.  I am or was involved in
 Creation and operation of the McMaster Health Forum (MHF)
 Development and maintenance of Health Systems Evidence (HSE)
 Development of (soon-to-launch) Health Systems Learning (HSL)
 Work of WHO-sponsored Evidence-Informed Policy Networks
(EVIPNet)
 Development of the SUPPORT tools
Conflict of Interest Statement
2

3.  Rationales for and a definition of evidence-informed decision-making
 Promising practices in addressing the challenges to finding and using
research evidence
 What is needed to do this work, with a detour about
 Clarifying a problem
 Framing options
 Identifying implementation considerations
 My top three priorities for action
Overview
3

4.  Which would be considered the most compelling in NSW / Australia?
 Better process / more efficient process?
 Better decisions / more acceptance of decisions?
 Better communication of decisions / less embarrassment?
 Better outcomes / better value for money?
 Note that research evidence can be used in many ways
 Solve particular problems at hand (instrumental)
 Think about problems/options differently (conceptual)
 Justify a decision made for other reasons (political)
Possible Rationales for
Evidence-Informed Decision-making
4

5.  Using the best available* data and research evidence – systematically
and transparently – in the time available in each of
 Agenda setting (esp. clarifying the problem iteratively, while being
attentive to policy and politics)
 Policy or program development (esp. framing options iteratively)
 Implementation (esp. identifying barriers / facilitators iteratively and
strategies to address them)
* Best available research evidence = highest quality, most locally
applicable, synthesized research evidence (looking first for a perfect
match to support an instrumental use and then looking more broadly to
support a conceptual use)
A Possible Definition of
Evidence-Informed Decision-making
5

6. 1. Research evidence competes with many other factors in the
decision-making process
2. Research evidence isn’t valued as an information input
3. Research evidence isn’t relevant
4. Research evidence isn’t easy to use
Caveat
 The approaches I’ll describe are promising (based on a
synthesis of findings from 124 observational studies) but not yet
tested rigorously in terms of effectiveness
Addressing the Challenges in Using
Research Evidence
6

7. Challenge 1
 Research evidence competes with many other factors in the
decision-making process
• Institutional constraints
• Interest group pressure
• Other ideas such as citizens’ values, tacit knowledge,
real-world views and experiences
• External events (e.g., global recession)
One option (among many) for addressing challenge 1
 Improve decision-making processes (but this is beyond the
scope of many of us) or create “routine” processes (as many
countries have done for new technologies) (e.g., NICE)
Addressing Challenge 1
7

8. Challenge 2
 Research evidence isn’t valued as an information input
Two options (among many) for addressing challenge 2
 Convince decision-makers and stakeholders to place value
on the use of research evidence by highlighting examples
from the past or from other jurisdictions where research
evidence made the difference between policy/program
success and failure (or between communication success
and failure) (e.g., WHO)
 Work with journalists to highlight statements that are and
aren’t based on the best available research evidence (e.g.,
Science-ish)
Addressing Challenge 2
8

9. Challenge 3
 Research evidence isn’t relevant
One option (among many) for addressing challenge 3
 Engage decision-makers and stakeholders periodically in priority-
setting processes and communicate the priorities to researchers
• Long-term requirements for data-collection systems and for
new primary research (e.g., Listening for Direction)
• Medium-term term requirements for systematic reviews (e.g.,
3ie)
• Short-term requirements for ‘evidence briefs’ (e.g., EVIPNet
Cameroon)
Addressing Challenge 3
9

10. Challenge 4
 Research evidence isn’t easy to use
Challenge 4a
 Research evidence isn’t communicated effectively
(i.e., decision-makers and stakeholders hear noise instead of
music)
Two options (among many) for addressing challenge 4a
 Identify a high-priority issue, identify data and systematic
reviews that address different facets of the issue (problem,
options and implementation considerations), and prepare an
‘evidence brief’ to summarize the findings (e.g., EVIPNet, MHF)
 Stop (incentivizing) researchers and university media offices
from publicizing the results of single studies without putting them
in the context of a systematic review (and clarifying the many
other questions that need to be answered before action is taken)
Addressing Challenge 4a
10

11. 11
Addressing Challenge 4a (2)

12. Challenge 4
 Research evidence isn’t easy to use
Challenge 4b
 Research evidence isn’t available when decision-makers and
stakeholders need it and in a form that they can use
Three options (among many) for addressing challenge 4b
 Promote the use of ‘one-stop shops’ for synthesized research
evidence
• e.g., Cochrane Library and PubMed ‘hedges’ for ‘my health’
• e.g., Health Evidence and PubMed ‘hedges’ for ‘our health’
• e.g., Health Systems Evidence for ‘our system’
 Staff a rapid-response unit (e.g., REACH Uganda, MHF)
 Provide training (and tools) for decision-makers and stakeholders
about how to find and use research evidence (e.g., MHF’s Health
Systems Learning)
Addressing Challenge 4b
12

13. Challenge 4
 Research evidence isn’t easy to use
Challenge 4c
 Decision-makers lack mechanisms to prompt them to use
research evidence in decision-making
One option (among many) for addressing challenge 4c
 Propose changes to cabinet submissions and senior
management committee briefings (and to periodic program
reviews) to prompt analysts to summarize whether and how
research evidence informed the clarification of a system
problem, the framing of options to address the problem, and the
proposed approach to implementation (e.g., Ontario’s Research
Evidence Tool)
Addressing Challenge 4c
13

14. Challenge 4
 Research evidence isn’t easy to use
Challenge 4d
 Decision-makers lack forums where challenges can be
discussed with stakeholders and researchers
One option (among many) for addressing challenge 4d
 Plan deliberative dialogues at which pre-circulated evidence
briefs serve as the starting point for off-the-record deliberations
involving decision-makers, stakeholders and researchers
(stakeholder dialogues) or exclusively citizens (citizen panels)
(e.g., EVIPNet, MHF)... A big step beyond GOBSATT
Addressing Challenge 4d
14

15. A Potential Two-Pronged Approach
Approach 1 – What researchers and ‘knowledge brokers’ can do
Prepare evidence / citizen briefs and convene stakeholder dialogues /
citizen panels over time frames of weeks and months
Prepare rapid syntheses over time frames of days and weeks
Promote one-stop shops and provide training so decision-makers and
stakeholders can find research evidence on their own over time frames of
hours and days
Approach 2 – What decision-makers can do
Send clear signals that using research evidence is a high priority
Establish a performance requirement for staff
Establish an internal rapid-response unit
Establish a requirement to summarize whether and how research
evidence informed understanding of the problem, options and
implementation considerations before decisions are taken
15

16. What Is Needed To Do This Work
Knowledge (see the ‘summary sheet’)
Questions to ask about a problem, options and implementation
considerations
Types of research evidence needed to answer these questions
Appropriate sources of key types of research evidence
What an AMSTAR score means
Questions to ask about local applicability considerations
Difference between knowing what other states or countries are doing
(jurisdictional scans) versus knowing the results of research conducted in
other countries (research evidence)
16

17. What Is Needed To Do This Work (2)
Attitudes
Working iteratively to understand a problem, options and
implementation considerations in light of institutional constraints,
interest group pressure, values and many other types of information, as
well as ‘external’ factors such as the state of the economy
Being systematic and transparent in finding and using research
evidence as one input to the decision-making process
Finding and using the best available (i.e., highest quality, most locally
applicable, synthesized) research evidence in the time you’ve been
given
Looking first for a perfect match in the available research evidence (to
support an instrumental use) and then looking more broadly (to support
a conceptual use)
17

18. What Is Needed To Do This Work (3)
Skills (for the ‘doers’)
Clarifying a problem (and its causes), framing options to address the
problem, and identifying implementation considerations
Searching appropriate sources of research evidence
Using AMSTAR to assess the quality of a systematic review
Conducting a local applicability assessment
Commissioning research to fill gaps in research evidence
18

19. 1. What is the problem (and its causes)?
2. How did the problem come to attention and has this process influenced
the prospect of it being addressed?
3. What indicators can be used, or collected, to establish the magnitude
of the problem and to measure progress in addressing it?
4. What comparisons can be made to establish the magnitude of the
problem and to measure progress in addressing it?
5. How can a problem be framed (or described) in a way that will motivate
different groups?
Clarifying the Problem - Questions
19

20.  Low rates of childhood immunization
 A risk factor, disease or condition – Re-emergence of some
preventable childhood diseases, such as measles
 A program, service or drug currently being used – Program coverage
rate dropped 5% in the past year and the program has not corrected
a celebrity’s assertions about safety and effectiveness
 Current health system arrangements within which programs, services
and drugs are provided
• Governance arrangements – Fragmented accountability
• Financial arrangements – Refugees have to pay out-of-pocket for
immunizations for their children
• Delivery arrangements – Many children are not registered with a
primary care team that can proactively reach them
 Current degree of implementation of an agreed course of action (e.g., a
policy) – Some parents are not aware of new schedule
Q1: What Is the Problem
(and its Causes)? Example 1
20

21.  Many citizens do not have a primary care physician
 A risk factor, disease or condition – Rising rates of chronic diseases
 A program, service or drug currently being used – Many citizens do not
benefit from coordinated prevention and proactive management of
chronic diseases
 Current health system arrangements within which programs, services
and drugs are provided
• Governance arrangements – No mechanism to introduce foreign
physicians without them ‘competing’ with local physicians
• Financial arrangements – Most physicians are paid on a fee-for-
service basis and can thrive even in well-served communities
• Delivery arrangements – Many physicians work in solo practices
 Current degree of implementation of an agreed course of action (e.g., a
policy) – Policymakers do not enforce return-of-service agreements
signed by physicians
Q1: What Is the Problem
(and its Causes)? Example 2
21

22.  Problem clarification involves
 Brainstorming about a problem and its potential causes (question 1)
 Searching for relevant data (questions 2 and 3) and research
evidence (questions 4 and 5) in appropriate sources (PubMed for
single studies or Health Systems Evidence for systematic reviews
related to health system arrangements)
• See the summary sheet
 Iteratively clarifying the problem (and its causes) in light of the data
and research evidence found
Clarifying the Problem - Summing Up
22

23. 1. What is an appropriate set of options to address the problem?
2. What benefits are important to those who will be affected and which
benefits are likely to be achieved with each option?
3. What harms are important to those who will be affected and which
harms are likely to arise with each option?
4. What are the local costs of each option and is there local evidence
about their cost-effectiveness?
5. What adaptations might be made to any given option and might they
alter its benefits, harms and costs?
6. Which stakeholders’ views and experiences might influence the
acceptability of an option and its benefits, harms, and costs?
Framing Options - Questions
23

24.  To address the problem of low rates of childhood immunization
1. Establish accountability among primary care practices for
registering all children in their catchment area and for achieving a
target immunization coverage rate (governance arrangement)
2. Remove all out-of-pocket charges for childhood immunization
(financial arrangement)
3. Undertake a mass-media campaign to correct a celebrity’s
assertions about the safety and effectiveness of childhood
immunization (program change) and to raise awareness about a
new immunization schedule (implementation strategy)
Q1: What is an Appropriate Set of
Options? Example 1
24

25.  To address the problem of many citizens not having a primary care
physician
1. Increase the supply of physicians by raising medical school
enrolment (delivery arrangement)
2. Improve the distribution of physicians by enforcing return-of-service
agreements signed by physicians (implementation strategy)
3. Improve the supply and distribution of physicians by introducing
geographically restricted licences (governance arrangement)
4. Change the dominant physician-remuneration mechanism from fee-
for-service payment to capitation (financial arrangement)
Q1: What is an Appropriate Set of
Options? Example 2
25

26.  What benefits are important to those who will be affected and which benefits are
likely to be achieved with each option? (Question 2)
 Systematic reviews of effectiveness studies (e.g., randomized trials)
 What harms are important to those who will be affected and which harms are likely
to arise with each option? (Question 3)
 Systematic reviews of effectiveness studies or observational studies
 What are the local costs of each option and is there local evidence about their cost-
effectiveness? (Question 4)
 Local cost-effectiveness analysis
 What adaptations might be made to any given option and might they alter its
benefits, harms and costs? (Question 5)
 Systematic reviews of qualitative studies (process evaluations)
 Which stakeholders’ views and experiences might influence the acceptability of
an option and its benefits, harms, and costs? (Question 6)
 Systematic reviews of qualitative studies (e.g., acceptability studies)
Research Evidence Can Help to Respond
to the Additional Questions
26

27.  Options framing involves
 Brainstorming about an appropriate set of options to address a
problem (question 1)
 Search for research evidence (in the Cochrane Library + PubMed,
Health Evidence + PubMed, or Health Systems Evidence) about
each option in turn
• See the summary sheet
 Iteratively framing the options in light of the research evidence found
Framing Options - Summing Up
27

28. 28
Identifying Implementation
Considerations - Questions to Consider
1. Where are the potential barriers to (and facilitators of) the successful
implementation of an option? Are they at the level of
o Patients/citizens (see question 2)
o Health workers (see question 3)
o Organizations (see question 4)
o System (see question 5)
1. What strategies should be considered in order to facilitate the
necessary behavioural changes among patients/citizens?
2. What strategies should be considered in order to facilitate the
necessary behavioural changes among health workers?
3. What strategies should be considered in order to facilitate the
necessary organizational changes?
4. What strategies should be considered in order to facilitate the
necessary system changes?

29.  Identifying implementation considerations involves
 Brainstorming about (and searching for local data and research
evidence about) potential barriers to (and facilitators of) the
successful implementation of an option (question 1)
 Searching Health Systems Evidence for research evidence about
each potential strategy in turn
• Benefits (question 2)
• Harms (question 3)
• Cost-effectiveness (question 4)
• Adaptations (question 5)
• Stakeholders’ views and experiences (question 6)
 Iteratively calibrating the strategies in light of the research evidence
found
Identifying Implementation
Considerations - Summing Up
29

30. My Top Three Priorities for Action
 Establish a requirement to summarize whether and how research
evidence informed understanding of the problem, options and
implementation considerations before decisions are taken
 Promote training for (and the use of tools and resources by) decision-
makers and stakeholders so they can find and use the best available
data and research evidence – systematically and transparently – in the
time available
 Create incentives for and communicate priorities to those who have the
knowledge, attitudes and skills to
 Prepare rapid syntheses in days and weeks
 Prepare evidence briefs (or citizen briefs) and convene stakeholder
dialogues (or citizen panels) in weeks and months
(and stop incentivizing researchers and university media offices to
publicize the results of single studies without clear justification)
30

31. Resources
 Available on Health Systems Evidence (www.healthsystemsevidence.org)
 Summary sheet on ‘finding and using research evidence’ (one page)
 Hyperlinked list of SUPPORT tools (two pages)
 Health Systems Evidence (four pages)
31