2. Outline
Introduction to critical appraisal
Appraising quantitative research
Levels of evidence
Statistics
Group appraisal
Appraising qualitative research
Qualitative methodologies
Group appraisal
Running a critical appraisal workshop
4. Evidence based healthcare
“When we intervene in the lives of
others we should do so on the basis
of the best evidence available
regarding the likely consequences
of that intervention”
G Macdonald, 1998
Macdonald, G. (1998). Promoting Evidence-Based Practice in Child Protection.
Clinical Child Psychology and Psychiatry, 3 (1), 71–85.
5. Steps of evidence based healthcare
1. The patient presents with a clinical
problem
2. Formulate a focused research question
3. Search for the best evidence
4. Critically appraise the evidence you
find
5. Consider the evidence in the light of
your expertise and decide whether to
apply it or not
7. Why critically appraise?
• In order to keep up to date,
clinicians would have to read
17 articles a day, 365 days a
year
• Research is of variable
quality
• Only an estimated 1% is
judged clinically relevant
• Need to find the 1%
8. How it works
• Involves answering a short
questionnaire
• We use the CASP questionnaires at
http://www.casp-uk.net/
• The questionnaires were devised by
clinicians for clinicians
• Remember to focus on appraisal not
criticism
10. What is quantitative research?
• Research on something that can be
accurately measured
• Tends to use large, statistically
representative samples
• Uses statistical methods to analyse
data
• Represents findings with numbers
• Attempts to eliminate bias
11. Systematic reviews
• A review of all the literature on a topic
• Systematically identified
• Appraised
• Summarised
18. Question 3
Do you think the important
relevant studies were included?
19. Publication bias
Papers with “interesting” results are more
likely to be:
• Submitted and accepted for publication
• Published in a major journal
• Published in English
• Quoted by authors
• Quoted in newspapers
20. Question 4
Did the review’s authors do
enough to assess the quality of
the included studies?
22. Quality assessment
• Were the inclusion/exclusion criteria
clear?
• Did they use any sort of scoring
system?
• Were the studies for inclusion
assessed by more than one reviewer?
23. Question 5
If the results of the review have
been combined, was it reasonable
to do so?
24. Heterogeneity
• Are the studies similar in terms of:
– Population
– Intervention
– Outcome
– Methodology
• Are the results similar from study to
study?
• Were any tests for heterogeneity
carried out?
• Are there any discussions around
heterogeneity?
38. P-value
Could the result have
occurred by chance?
p = 0.001 (1 in 1000)
p = 0.2 (1 in 5)
A p-value of less than 0.05 (1 in 20) is
considered to be statistically significant
40. Application
• Difficult to answer if you don‟t have
your own population
• Need to rely on experts
• Think about whether the study could be
easy replicated with another population
42. Outcomes
• Do the authors address all the
outcomes they set out to study?
• Are the outcomes considered from
different viewpoints e.g. patient,
practitioners, policy makers?
44. Benefit v harms and costs
• How significant are the
benefits?
• Is there any discussion of
possible side effects etc.?
• Are financial costs
mentioned?
47. What is qualitative research?
• Concerned with the world of human
experience from the view of the
participants
• Natural settings are used
• A study is „led‟ by the subjects‟
experiences
• Qualitative research is rigorous and
systematic
• Results are not usually generalisable or
totally replicable
50. The research question
• Describes why the research is being
carried out
• Qualitative research addresses the
questions
• Are the aims of the research clear?
52. Why qualitative research?
• Increases knowledge in an area that is
poorly understood
• Challenges assumptions and practices
• Acts as a precursor to quantitative
research
• Generates new ideas
53. Question 3
Was the research design
appropriate to address the aims of
the research?
55. Question 4
Was the recruitment strategy
appropriate to the aims of the
research?
56. Sampling
• Sample size can be determined by
– Data analysis
– Time constraints
• What population does the sample refer
to?
• How was the sample selected?
67. The findings
• Are the findings explicit?
• Are they credible?
– respondent validation
– quality of argument
– would another researcher make a similar
interpretation?
– are alternative interpretations explored?
70. Value of the research
Consider your appraisal of the paper in
terms of:
• Credibility
• Dependability
• Transferability
• Confirmability
How useful is the paper to you?
73. Before the session
• Who is the group?
• Is it qualitative, quantitative, or both?
• Know what equipment and facilities are
available
• Ask participants to read the article in
advance
74. Structuring the session
• Dependent on the group and time frame
• Presentation
• Encourage audience participation
• Use interactive activities
• Practical critical appraisal in groups
• Feedback
75. Choosing an article
• For a mixed group choose a fairly generic
article
• For a discipline specific group:
– Do you want to use a discipline specific
article?
– Ask for topic ideas
– Not a good idea to let them choose the
articles
• Choose studies with both strengths and
weaknesses
• Make sure it‟s easily available to all the
participants