This document provides an introduction to critical appraisal of literature. It discusses the importance of critically evaluating research to separate reliable evidence from unreliable evidence. It outlines the process of critical appraisal, including asking a focused question, finding relevant evidence, and using appraisal tools to systematically examine research quality, validity, and relevance. The document also introduces some key statistical concepts used in research, such as p-values, confidence intervals, risk reduction, and number needed to treat. The goal of critical appraisal is to make informed decisions about integrating research findings into clinical practice and policy.

1. Introduction to
Critical Appraisal
Dominic Gilroy
Library & Knowledge Service Manager
Leeds & York Partnerships NHS FT

2. Objectives
• Why we need to evaluate literature
• How to form a question that will result in a
targeted literature search
• Critical appraisal of the evidence – using
appraisal tools/checklists
• Thoughts on statistics

3. What is Critical Appraisal
“Critical appraisal is the process of carefully and systematically
examining research to judge its trustworthiness, and its value
and relevance in a particular context”
Burls (2009) What is critical appraisal?
http://www.medicine.ox.ac.uk/bandolier/painres/download/whatis/What_is_critical_appraisal.pdf
More than just reading and summarising a paper.

4. Critical Appraisal is:
• Balanced assessment of strengths of research against its weaknesses
• Assessment of research process and results
• Consideration of quantitative and qualitative aspects of research
• To be undertaken by all health professionals as part of their work
as just one part of the decision making process

5. Critical Appraisal is not:
• Negative dismissal of a piece of research
• Assessment of results alone
• Based entirely on detailed statistical analysis
• To be undertaken by expert researchers only

6. Why bother?
• Published evidence is not always reliable –
a way of separating “wheat from the chaff”
• Only an estimated 2% is judged clinically relevant
• Patients are better informed and increasingly present
having read material on the web or in the media

7. Challenges:
• Can be time-consuming
• It does not always give you an easy answer – there are still
judgements and decisions to be made
• It may not give the expected or hoped for result
• It can be dispiriting if it highlights a lack of good evidence
– need determination to persist with an area of interest
when access to good research in the area is limited
• Maths and statistics …..!

8. When to do Critical Appraisal
Ask the
question
Find the
evidence
Appraise and
interpret the
evidence
Act on appropriate
evidence
Evaluate and
reflect

9. Step 1: Asking the question
A well structured question is the key to finding evidence
Contains 4 parts (usually) … PICO
PICO
P = patient or problem
How would you describe a group of patients similar to yours?
I = intervention or treatment
What is the main action you are considering?
C = comparison or alternative
What is/are the other option(s) available?
O = outcome
What do you/the patient want to happen?

10. Step 2: Finding the evidence
IDENTIFICATION OF THE BEST AND
MOST RELEVANT RESOURCE(S) TO
SEARCH SHOULD START FROM THE
TOP OF THE EVIDENCE HIERARCHY
Systematic
Reviews
Expert Opinion
Randomised
Controlled
Trials

11. Anatomy of a paper
Look for:
Abstract - abbreviated form of the study
Introduction - problem being studied and explains why the research is being
carried out
Methods - information concerning the experiment design, including study
sample, treatment allocation, outcome measures
Results - clear presentation of the experiments results, including any adverse
effects or variables that may have affected the outcome
Discussion/conclusion - gives the author a chance to give their viewpoint
and explain any clinical importance of the experiment

12. Appraisal Tools
• Several different appraisal tools are available
• Check sheets of question
• Different tools for different types of research
• Helps make appraisal quick, simple, effective and consistent
• Can help get rid of personal bias
• Help identify if results are significant and valid

13. Implications of the research:
Consider:
• Does it produce new knowledge?
• How applicable is the research to my own patients?
Perhaps the ultimate question …
• Should you act on the findings and change practice?

14. Practice Paper
1. Read the paper
2. Look at checklist points 1-6

15. Practice Paper
Feedback

16. Some numbers and statistics
• Presentation of statistics
• Sample Size
• P Values
• Confidence Intervals
• Risk Reduction
• Number Needed to Treat
Disclaimer:
(Most) Librarians are not statisticians
This session is not about in depth-
statistics but about how to interpret
statistics when reading the paper

17. Presentation of Statistics
• The same statistics can be presented in numerous ways
• The way in which statistics are presented can affect how we
interpret the results
• Authors may use this when writing papers to suggest a
particular view of the results

18. Shock Horror!
Daily Mail Online
Why eating just one
sausage a day increases
your risk of cancer by 20%
http://www.dailymail.co.uk/health/article-550729/Why-eating-just-
sausage-day-raises-cancer-risk-20-cent.html

19. Critically Appraised
Why eating just one sausage a day raises your
cancer risk by 20 per cent
• Specifics – This refers to Bowel Cancer
• Baseline – the usual risk for bowel cancer in a lifetime is:
- 1 in 18 for women (5.6%)
- 1 in 20 for men (5%)
• The risk with a “sausage a day” is not 25% (5% + 20%) or 1 in 4
• The risk with a “sausage a day” increases by 20% of the
original risk (20% of 5% = 1%)
So the new risk if you eat a sausage is a far less dramatic:
- 1 in 15 for women (6.6%)
- 1 in 17 for men (6%)

20. Which would you prefer to fund?
The following are four different rehabilitation programmes for heart attack
victims:
Programme A – which reduced the rate of deaths by 20%
Programme B – which produced an absolute reduction in deaths of 3%
Programme C – which increased patients’ survival rate from 84% to 87%
Programme D – which meant that 31 people needed to enter the
programme to avoid one death

21. Which would you prefer to fund?
The following are four different rehabilitation programmes for heart attack
victims:
Programme A – which reduced the rate of deaths by 20%
Programme B – which produced an absolute reduction in deaths of 3%
Programme C – which increased patients’ survival rate from 84% to 87%
Programme D – which meant that 31 people needed to enter the
programme to avoid one death
They all refer to the same study!
(Taken from: How to read a paper / Trisha Greenhalgh, 2010, 4th ed.)

22. Sample Size
“A trial should be big enough to have a high chance of detecting…
a worthwhile effect if it exists, and thus to be reasonably sure that
no benefit exists if it is not found in the trial”
Practical Statistics for Medical Research/ Doug Altman, 1991
• A sample size may be given in the paper
• Don’t worry about how they have calculated it but……
• If they do provide a sample size check if they actually achieve
the stated number in their recruiting figures
• If they do not provide the sample size look at the numbers
included in the trial as low numbers can mean a less reliable study

23. P Values
Are useful in deciding if a result could be just by chance:
0 1
Impossible that result is by chance Certainty that result is by chance
(Good) (Bad)
P= 0.001 Very unlikely 1 in 1000
P= 0.05 Fairly unlikely 1 in 20
P= 0.5 Fairly likely 1 in 2
P = 0.75 Very likely 3 in 4
A P value of <0.05 is generally considered statistically significant
If P <0.05 then the result in question is likely to be more than just chance

24. Confidence Intervals #1
“A confidence interval….allows you to estimate for both positive trials and
negative ones whether the strength of the evidence is strong or weak.”
Greenhalgh, Trisha (2014) How to read a paper
If the same trial was run many, many times you might not expect the exact
same results every time – but the results should lie within a specific range.
With a 95% CI we are saying that 95% of the time the results lie within a
specific range.
Trial result
Range of results of 95%
Exceptions to the rule (outliers)
(The other 5%)

25. Confidence Intervals
Commute Example
John’s journey home
Average = 50 minutes
Can often take up to 55 minutes
Can be as quick as 45 minutes
Very occasionally (5% of time or less) it can take over an hour
(eg: traffic accidents etc.) or can be quicker if he travels at off peak times
John’s journey time could be expressed as: 50 min (95% C.I. 45 min – 55 min)
Avishai Teicher via the PikiWiki - Israel free image collection project

26. Confidence Intervals #2
The range of the confidence interval (CI) is important.
The narrower it is the more accurate and precise the results
The broader it is the wider the possibility and range of likely results
Referring back to our commute example:
It is more useful to tell someone it may take between 45 and 55 minutes to travel somewhere than it would
be to say it could take between 30 minutes and 2 hours. It would help them to plan better.
Precision is usually preferable.

27. Risk Reduction
Absolute Risk Reduction (ARR)
If an intervention decreases risk of an event occurring from
2/100 to 1/100
1% absolute risk reduction
Relative Risk Reduction (RRR)
If an intervention decreases risk of an event occurring from
2/100 to 1/100
50% relative risk reduction

28. Number Needed to Treat (NNT)
For every n patients treated with the intervention, one extra
patient would be expected to benefit
i.e. how many people do you need to treat to be certain that one
person would benefit
Example: If NNT = 10 then on average for every ten patients
treated we would expect one of them to benefit

29. Number Needed to Treat (NNT)
Extreme examples:
For the use of antibiotics Cimetidine, Ranitidine & Omeprazole to
eradicate Heliobacter Pylori from the stomach
NNT = 1.3
For the use of cholesterol lowering Statins in the primary
prevention of vascular events like stroke and heart disease
NNT = 1000

30. Useful Calculations
Absolute Risk Reduction (ARR) = Control Rate – Experimental Rate
Relative Risk Reduction (RRR) = Absolute Risk Reduction / Control Rate
Number Needed to Treat (NNT) = 100 / Absolute Risk Reduction
Control Rate = Chance of outcome in control group (%)
Experimental Rate= Chance of outcome in experimental group (%)

31. Example Exercise
Infected with flu? Yes No Total
New Trial Drug 38 73 111
Placebo (control) 79 35 114
Total 117 108 225
Outcome: Preventing influenza
Control Rate = Chance of outcome in control group (%)
(79/114) x 100 = 69%
Experimental Rate= Chance of outcome in experimental group (%)
(38/111) x 100 = 34%

32. Example Exercise (continued)
Outcome: Preventing influenza
Absolute Risk Reduction (ARR) = Control Rate – Experimental Rate
69 - 34 = 35%
Relative Risk Reduction (RRR) = Absolute Risk Reduction /Control Rate
35 / 69 = 0.5%
Number Needed to Treat (NNT) = 100 / Absolute Risk Reduction
100/35 = 2.85 round up to 3
Control Rate = 69%
Experimental Rate=34%

33. Practice Paper Part 2
Look at the checklist and try to respond to points 7 - 11
• What is the sample size and do they achieve it?
• Are there any P values given in the results and what do these suggest?
Looking at the primary outcome measure
• What is the absolute risk reduction ?
• What is the relative risk reduction ?
• What is the number needed to treat?

34. Useful Websites
Critical appraisal
http://www.casp-uk.net/
http://jamaevidence.com/
Stats
http://www.medicine.ox.ac.uk/bandolier/glossary.html

35. Further Reading
Greenhalgh, Trisha (2014) How to read a paper: the basics of evidence-
based medicine. Oxford: Wiley Blackwell. 5th ed.
Crombie, Iain K (1996) Pocket guide to critical appraisal.
London: BMJ.
Straus, Sharon (ed.) (2005) Evidence-based medicine: how to practice and
teach EBM. Edinburgh: Elsevier Churchill Livingstone. 3rd ed.
Ajetunmobi, Olajide (2002) Making sense of critical appraisal.
London: Arnold.

36. Conclusion
We have examined:
• Why we need to evaluate literature
• Critical appraisal of the evidence –
using appraisal tools / checklists
• Statistics introduction

37. Contact Details
Dominic Gilroy
Library & Knowledge Service Manager
Leeds & York Partnerships NHS FT
dominic.gilroy1@nhs.net
0113 85 55658
Leeds Libraries for Health
www.leedslibraries.nhs.uk