This document outlines an agenda for a class on critically appraising medical literature. It begins with reviewing the health science research process and types of research studies. It then discusses what a critical appraisal is and how to appraise a research paper. Key questions to ask when appraising medical literature are presented. The document reviews levels of evidence and study types like randomized controlled trials, cohort studies, and case-control studies. It provides guidance on critically appraising diagnostic studies, intervention studies, and calculating p-values. An in-class appraisal activity is planned to conclude the session.

1. PT650 1A
Scientific Inquiry I:
Critical Appraisal of Medical
Literature
Linda Galloway
lgallowa@chapman.edu
Health Sciences Librarian, Rinker Health Science Campus
June 30, 2016

2. Today’s Agenda
• Review of health science research process
• Types of research studies and levels of evidence
• Critical appraisal of research studies
• What is a Critical Appraisal?
• How to appraise a research paper
• In-class appraisal activity

3. ASK ACQUIRE APPRAISE
Formulate a researchable question, find relevant articles,
and evaluate articles for suitability
Review of the health science
research process…

4. ASK: Formulate a focused
question
Patient / Problem / Population
Intervention
Comparison
Outcome
PICO is a mnemonic that helps one remember the key
components of a well focused question. The question needs to
identify the key problem of the patient, what treatment or tests
you are considering for the patient, what alternative treatment
or tests are being considered (if any) and what is the desired
outcome to promote or avoid.

5. Our PICO
A sedentary 82 year old woman living alone has been falling frequently. Her
doctor has referred her to PT to help strengthen her legs and improve her
balance.
Patient / Problem / Population – 82 year old white female who is falling
Intervention – physical therapy, movement exercises
Comparison – do nothing (modify home environment)
Outcome – prevent falls at all times of day
Can physical therapy interventions reduce falls in elderly patients?

6. Developing a search strategy
1. Frame the question – use PICO
2. Find background information – use
AccessMedicine or 5Minute Consult or…
3. Develop a search strategy using keywords and
phrases
4. Perform PubMed search, and refine search
terms as you retrieve results
5. Use the tools in the database to help with your
search!
6. Evaluate the results obtained, and further
refine if necessary

7. ASK ACQUIRE APPRAISE
Find relevant articles

8. PubMed Advanced Search – choosing ‘Publication Type’

10. ASK ACQUIRE APPRAISE
Evaluate articles for suitability
Critical Appraisal of
Research Topics

11. Critical Appraisal Learning
Outcomes
By the end of this section you should:
•Understand the principles of critical
appraisal and why you should undertake it
•Be able to appraise published research and judge
its reliability
•Be able to assess the relevance of published
research to your own work

12. What is Critical Appraisal?
‘Critical appraisal is the assessment of evidence by
systematically reviewing its relevance, validity and
results to specific situations’
Chambers, R (1998)
A balanced assessment of strengths of research against its
weaknesses. It is a skill that needs to be practiced!

13. Before we appraise medical
studies, let’s review levels of
evidence and study types

14. Dartmouth Biomedical Libraries: Resources.
http://www.dartmouth.edu/~biomed/resources.htmld/guides/ebm_resources.shtml.
Accessed April 29, 2016
Quality of evidence

15. Study Types Defined
meta-analysis: a quantitative approach in which individual study findings addressing a
common problem are statistically integrated and analyzed to determine the
effectiveness of interventions.
systematic review: a process by which a body of literature is reviewed and assessed
using systematic methods which are intended to reduce bias in the review process
and improve understandability.
randomized controlled trial: an experiment in which investigators randomly allocate
participants into (e.g. treatment and control) groups to receive or not to receive
one or more interventions that are being compared.
controlled clinical trial: any study which compares two groups by virtue of different
therapies or exposures.
cohort study: an observational study in which a cohort is followed over time.
case-control studies: retrospective research design that compares individuals with a
specific condition to those without it.
case report: a detailed report of the symptoms, signs, diagnosis, treatment, and
follow-up of an individual patient.
case series: a report on a series of patients with an outcome or condition of interest.
Susan Fowler, MLIS Washington University in St. Louis 2012

16. Levels of Evidence in
Medical Studies
Systematic
Reviews and Meta-
analyses
Randomised Controlled
Trials (RCTs)
Cohort Studies
Case Control Studies

17. Case-Control Studies
•Conducted after outcome has occurred
•Used mainly for causation studies
•Typically retrospective (looks back in time)
•Patient with outcome matched to control
•Investigations made into possible causes in both
patients
•May be only option in rare conditions
Example: Study used a matched design, matching infants
who had persistent pulmonary hypertension with infants
who did not have it, and compared the rates of exposure
to SSRIs.

18. Cohort Studies
• Two or more populations (cohorts) are followed
prospectively to see which risk factors are associated
with respect to disease or outcome
• One group has received an intervention or exposure
• Groups otherwise closely matched
• Groups followed over time
• Can be used for causation, diagnostic, harms and
therapeutic studies
Example: To determine the long-term effectiveness of influenza vaccines in elderly
people, cohorts of vaccinated elderly and unvaccinated community-dwelling elderly
were studied. The results suggest that the elderly who are vaccinated have a
reduced risk of hospitalization for pneumonia or influenza.

19. Randomized Controlled
Trials (RCT)
•Treatment group and ‘control’ group
•Random assignment to groups
•May involve ‘blinding’ of participants and
researchers
•Used for therapeutic or diagnostic interventions
•Some interventions unsuitable for RCTs
•Expensive
Example: This research studied the effect of raloxifene on fracture risk
in postmenopausal women, and found that the women who took
raloxifene over the same five year period of time as the women who did
not reduced their risk of clinical vertebrate fracture.

20. SystematicReviewsand
Meta-Analyses
•Combination of results of several RCTs or other types
of evidence
•Meta-analyses includes a statistical combination of
the results
•Considered to be highest-level evidence if conducted
properly
•Relies on enough RCTs being available
•Can include ‘Odds-Ratio’ diagrams
Example: Respiratory muscle training increases respiratory
muscle strength and reduces respiratory complications after
stroke: a systematic review.

21. Other Types of Clinical
Research
These are lesser forms of evidence, but for some
interventions, exposures or conditions they may be the only
form available
• Case studies / Case series - compares patients who have a
disease or outcome of interest (cases) with patients who
do not have the disease or outcome (controls), and looks
back retrospectively to compare how frequently the
exposure to a risk factor is present in each group to
determine the relationship between the risk factor and the
disease. These studies are designed to estimate odds.
• Cross-sectional surveys – Data collected once in a short
period of time

22. Quiz
What Type of clinical research is this?
1. Two groups of doctors, one group smokers, the
other non-smokers are followed over the course of
20 years to see whether which group are more
likely to develop lung cancer

23. Answer
Cohort study

24. Quiz
What Type of clinical research is
this?
2. Two groups of patients are studied, one
group given physiotherapy for low back pain,
the other given advice only. Patients are
randomly assigned to either group and
followed up after six months

25. Answer
Randomised controlled trial

26. Quiz
What Type of clinical research is this?
3. One hundred sets of twins, where one had
developed melanoma and the other had not,
were studied for possible causation factors

27. Answer
Case-control study
(As each twin should be closely matched in most
aspects other then one having developed
melanoma)

28. Break!

29. Critical Appraisalof MedicalResearch
•Balanced assessment of strengths of research
against its weaknesses
•Assessment of research process and results
•Consideration of quantitative and qualitative
aspects of research
•To be undertaken by all health professionals as
part of their work

30. Critical Appraisalis NOT:
• Negative dismissal of a piece of research
• Assessment of results alone
• Based entirely on detailed statistical
analysis
• To be undertaken by expert researchers
only

31. Why Should We Bother?
• Published research is not
always reliable or applicable
• Patients are informed by
the news and media
reporting the results of
scientific studies
Drinking more coffee may
stave off multiple
sclerosis!!
Gluten triggers
strange
delusions!
Low fat diet bad for your health
and cutting back on meat, dairy
and eggs a disastrous mistake!

32. Questions to ask
when appraising
medical literature

33. Appraising medical literature
1. Is the study question relevant? Study should address an
important topic, add to what is already known and be
relevant to one’s work.
2. Does the study add anything new? Scientific research is
built upon previous research – subsequent studies should
extend research and enhance knowledge.
3. What type of research question is being asked? A well-
developed research question usually identifies three
components: the group or population of patients, the
studied parameter (e.g. a therapy or clinical intervention)
and the outcomes of interest. Typically questions address:
Effectiveness of Treatment or the Frequency of Events.

34. Appraising medical literature
4. Was the study design appropriate for the research
question? The research question determines the most
appropriate study design. Randomized Controlled Trials
(RCT) are the gold standard for intervention studies.
Cohort studies are common for diagnostic tests. There
are many different types of studies.
5. Did the study methods address the most important
potential sources of bias? Bias in this context does not
mean investigator bias, but rather bias attributed to
chance (e.g. a random error) or to the study methods
(systematic bias). Systematic biases arise from how
participants were selected, how data was collected, or
through the researchers' analysis or interpretation.

35. Appraising medical literature
6. Was the study performed according to the original
protocol? Deviations from the planned protocol can
affect the validity or relevance of a study – often
planned participation is lower than anticipated or drop-
outs occur.
7. Does the study test a stated hypothesis? A hypothesis
is a clear statement of what the investigators expect the
study to find and states the research question in a form
that can be tested and refuted
8. Were the statistical analyses performed correctly?
Difficult to ascertain; read the Methods section, and pay
attention to how missing data are treated. Ideally, this
section should be understandable to non-statisticians.

36. Appraising medical literature
9. Do the data justify the conclusions? Are the
conclusions that the authors present reasonable on the
basis of the accumulated data? Be wary of
generalizations and size of populations studied.
10. Are there any conflicts of interest? Personal factors,
including financial, career and outcome-based conflicts.
This is a judgement call on the part of the reader.

37. Questions?

38. How to do Critical Appraisal
•Checklists are available that ask simple
questions
•There are different checklists for different
types of studies (systematic reviews, RCTs
etc)
•These checklists help you focus on the
most important aspects of the article

39. Key Steps in CriticalAppraisal
The checklists help us answer these questions
• What are the results? Are they significant?
• Are the results valid? How was the research
done?
• How will these results help me work with my
patients?

40. Diagnostic Research Studies
• PT Diagnosis process:
• Patient history
• Systems review
• Tests & measures
• Diagnostic tests have a range of values
• Should be compared to the Reference or Gold Standard test
• e.g. MRI compared to PT musculoskeletal test
• ‘Index test’ or ‘test of interest’ are the tests being investigated

41. Appraising Diagnostic Studies
• Appraising diagnostic literature
• Determining applicability – relevance of clinical population to your
patient
• Quality – Was there an independent, blind comparison with the gold
standard diagnostic tool? Were examiners unaware of results of gold
std test? Did all subject receive both tests?
• Results: Clinically useful statistics reported and interpreted?
• Sensitivity – clinical test’s accuracy in correctly identifying problem as
compared to gold standard.
• Specificity – test’s ability to correctly identify absence of problem (no
problem)
• Summarizing clinical bottom line of diagnostic study
• Is the test accurate and clinically relevant to PT practice?
• Will the results of this test (knowing it’s limitations) affect treatment plan
and help patient?

42. Sensitivity: clinical test’s accuracy in correctly
identifying a problem as established by the Gold
Standard.
Specificity: clinical test’s ability to correctly
identify the absence of a problem.

43. Intervention Studies
• The intervention (the preventative or therapeutic measure)
being tested is allocated by the investigator to a group of two
or more study populations
• Subjects are followed prospectively to compare the
intervention vs. the control (standard treatment, no treatment
or placebo)
• The main intervention study design is the randomized
controlled trial (RCT)

45. Appraising Intervention
Studies
• Inclusion and exclusion criteria should be clearly defined AND
applicable
• Realistic interventions and controls
• Outcome measures relevant to the clinical question and
conducted realistically
• Appropriate random assignment of participants
• Bias minimized?
• Randomization
• Recruitment of participants
• Minimal dropout rate
• Masking or blinding of researchers, evaluators, participants,
therapists

46. p-value
The P value, or calculated probability, is the probability of finding
the observed, or more extreme, results when the null hypothesis (H
0) of a study question is true – the definition of 'extreme' depends
on how the hypothesis is being tested.
p-value < 0.05
Can state with 95% confidence that the result is not due to chance
Most authors refer to statistically significant as P < 0.05 and
statistically highly significant as P < 0.001 (less than one in a
thousand chance of being wrong).

47. Diagostic Study AppraisalRubric
Yes No ? Comments
1. Is the entire spectrum of patients
represented in the studysample?
2. Is my patient represented inthis
spectrum ofpatients?
3. Was there an independentblind
comparison with a reference
(gold) standard of diagnosis?
4. Do the authors provide evidence of the
“quality” of the gold standard?
5. Were the diagnostic tests performed
by one or more reliable examiners who
were masked to the results of the
reference (gold) test? (requires that
some level of evidence of the
qualifications of the examiners is
provided)
6. Did all subjects receive both (indexand
gold standard tests) regardless of test
outcomes?
7. Was there minimal time delay
between the administration of the index
tests and the "gold"standard?

48. Yes No ? Comments
8. Was the diagnostic test interpreted
independently of all clinical information?
9. Do the authors provide clinically useful
measures of the diagnostic test’s utility,
for instance likelihood ratios (patient
centered) to support the Sensitivity and
Specificity (test centered)?
10. Is the test accurate (when compared
to the Goldstandard)?
11. is the test clinically relevantto
physical therapy practice?
12. Will the resultingpost-test
probabilities affect my management and
help mypatient?
Summary Comments (take home
message):
danielcip3
2013-11-01 22:21:13
--------------------------------------------
Summary Comments (take home
message):

49. Additional sources of
Checklists
Critical appraisal checklists can be downloaded
from:
Critical Appraisal Skills Programme
www.casp-uk.net
Centre for Evidence-Based Medicine
www.cebm.net/index.aspx?o=1157

50. The Good News Is…
Some healthcare research has already been
appraised
Cochrane Library
(good quality systematic reviews conducted
by the Cochrane Collaboration)
Database of Abstracts of Reviews of Effects
(critical appraisals of published systematic reviews)

51. In-class appraisal
activity
Link to myNCBI bibliography

52. Questions?
Contact Linda Galloway, lgallowa@chapman.edu