1. By Ellen Fineout-Overholt, PhD, RN,
FNAP, FAAN, Bernadette Mazurek
Melnyk, PhD, RN, CPNP/PMHNP,
FNAP, FAAN, Susan B. Stillwell,
DNP, RN, CNE, and Kathleen M.
Williamson, PhD, RN
Critical Appraisal of the Evidence: Part I
An introduction to gathering, evaluating, and recording the evidence.
This is the fifth article in a series from the Arizona State University College of Nursing and Health Innovation’s Center
for the Advancement of Evidence -Based Practice. Evidence-based practice (EBP) is a problem-solving approach to the
delivery of health care that integrates the best evidence from studies and patient care data with clinician expertise and
patient preferences and values. When delivered in a context of caring and in a supportive organizational culture, the
highest quality of care and best patient outcomes can be achieved.
The purpose of this series is to give nurses the knowledge and skills they need to implement EBP consistently, one
step at a time. Articles will appear every two months to allow you time to incorporate information as you work toward
implementing EBP at your institution. Also, we’ve scheduled “Chat with the Authors” calls every few months to provide
a direct line to the experts to help you resolve questions. Details about how to participate in the next call will be pub-
lished with September’s Evidence-Based Practice, Step by Step.
I
n May’s evidence-based prac- database’s own indexing lan- library subscription or those
tice (EBP) article, Rebecca R., guage, or controlled vocabulary, flagged as “free full text” by a
our hypothetical staff nurse, matched the keywords or syn- database or journal’s Web site.
and Carlos A., her hospital’s ex- onyms, those terms were also Others are available through in-
pert EBP mentor, learned how to searched. At the end of the data- terlibrary loan, when another
search for the evidence to answer base searches, Rebecca and Car- hospital library shares its articles
their clinical question (shown los chose to retain 18 of the 18 with Rebecca and Carlos’s hospi-
here in PICOT format): “In hos­ studies found in PubMed; six of tal library.
pitalized adults (P), how does a the 79 studies found in CINAHL; Carlos explains to Rebecca that
rapid response team (I) compared and the one study found in the the purpose of critical appraisal
with no rapid response team (C) Cochrane Database of System- isn’t solely to find the flaws in a
affect the number of cardiac ar­ atic Reviews, because they best study, but to determine its worth
rests (O) and unplanned admis­ answered the clinical question. to practice. In this rapid critical
sions to the ICU (O) during a As a final step, at Lynne’s rec- appraisal (RCA), they will review
three­month period (T)?” With ommendation, Rebecca and Car- each study to determine
the help of Lynne Z., the hospi- los conducted a hand search of • its level of evidence.
tal librarian, Rebecca and Car- the reference lists of each study • how well it was conducted.
los searched three databases, they retained looking for any rele- • how useful it is to practice.
PubMed, the Cumulative Index vant studies they hadn’t found in Once they determine which
of Nursing and Allied Health their original search; this process studies are “keepers,” Rebecca
Literature (CINAHL), and the is also called the ancestry method. and Carlos will move on to the
Cochrane Database of Systematic The hand search yielded one ad- final steps of critical appraisal:
Reviews. They used keywords ditional study, for a total of 26. evaluation and synthesis (to be
from their clinical question, in- discussed in the next two install-
cluding ICU, rapid response RAPID CRITICAL APPRAISAL ments of the series). These final
team, cardiac arrest, and un­ The next time Rebecca and Car- steps will determine whether
planned ICU admissions, as los meet, they discuss the next overall findings from the evi-
well as the following synonyms: step in the EBP process—critically dence review can help clinicians
failure to rescue, never events, appraising the 26 studies. They improve patient outcomes.
medical emergency teams, rapid obtain copies of the studies by Rebecca is a bit apprehensive
response systems, and code printing those that are immedi- because it’s been a few years since
blue. Whenever terms from a ately available as full text through she took a research class. She
ajn@wolterskluwer.com AJN ▼ July 2010 ▼ Vol. 110, No. 7 47

2. shares her anxiety with Chen M., new EBP team, Carlos provides and the Boston ­ niversity Medi-
U
a fellow staff nurse, who says R
­ ebecca and Chen with a glossary cal Center Alumni Medical Li-
she never studied research in of terms so they can learn basic brary [http://medlib.bu.edu/
school but would like to learn; research terminology, such as sam­ bugms/content.cfm/content/
she asks if she can join Carlos ple, independent variable, and de­ ebmglossary.cfm#R].)
and ­ ebecca’s EBP team. Chen’s
R pendent variable. The glossary Determining the level of evi-
spirit of inquiry encourages Re- also defines some of the study de- dence. The team begins to divide
becca, and they talk about the signs the team is likely to come the 26 studies into categories ac-
opportunity to learn that this across in doing their RCA, such cording to study design. To help
project affords them. Together as systematic review, randomized in this, Carlos provides a list of
they speak with the nurse man- controlled trial, and cohort, qual- several different study designs
ager on their ­ edical–surgical
m itative, and descriptive studies. (see Hierarchy of Evidence for
unit, who agrees to let them use (For the definitions of these terms Intervention Studies). Rebecca,
their allotted continuing educa- and others, see the glossaries pro- Carlos, and Chen work together
tion time to work on this project, vided by the Center for the Ad- to determine each study’s design
after they discuss their expecta- vancement of Evidence-Based by reviewing its abstract. They
tions for the project and how its Practice at the Arizona State Uni- also create an “I don’t know”
outcome may benefit the patients, versity College of Nursing and pile of studies that don’t appear
the unit staff, and the hospital. Health Innovation [http://nursing to fit a specific design. When they
Learning research terminol- andhealth.asu.edu/evidence-based- find studies that don’t actively
ogy. At the first meeting of the practice/resources/glossary.htm] answer the clinical question but
Hierarchy of Evidence for Intervention Studies
Type of evidence Level of evidence Description
Systematic review or I A synthesis of evidence from all relevant random­zed controlled trials.
i
meta-analysis
Randomized con­ II An experiment in which subjects are randomized to a treatment group
trolled trial or control group.
Controlled trial with­ III An experiment in which subjects are nonrandomly assigned to a
out randomization treatment group or control group.
Case-control or IV Case-control study: a comparison of subjects with a condition (case)
cohort study with those who don’t have the condition (control) to determine
characteristics that might predict the condition.
Cohort study: an observation of a group(s) (cohort[s]) to determine the
development of an outcome(s) such as a disease.
Systematic review of V A synthesis of evidence from qualitative or descrip­ive studies to
t
qualitative or descrip­ answer a clinical question.
tive studies
Qualitative or de- VI Qualitative study: gathers data on human behavior to understand why
scriptive study and how decisions are made.
Descriptive study: provides background informa­ion on the what,
t
where, and when of a topic of interest.
Expert opinion or VII Authoritative opinion of expert committee.
consensus
Adapted with permission from Melnyk BM, Fineout-Overholt E, editors. Evidence-based practice in nursing and healthcare:
a guide to best practice [forthcoming]. 2nd ed. Philadelphia: Wolters Kluwer Health/Lippincott Williams and Wilkins.
48 AJN ▼ July 2010 ▼ Vol. 110, No. 7 ajnonline.com

3. Critical Appraisal Guide for Quantitative Studies
1. Why was the study done?
• as there a clear explanation of the purpose of the study and, if so, what was it?
W
2. What is the sample size?
• ere there enough people in the study to establish that the findings did not occur by chance?
W
3. Are the instruments of the major variables valid and reliable?
• ow were variables defined? Were the instruments designed to measure a concept valid (did
H
they measure what the researchers said they measured)? Were they reliable (did they measure a
concept the same way every time they were used)?
4. How were the data analyzed?
• hat statistics were used to determine if the purpose of the study was achieved?
W
5. Were there any untoward events during the study?
• id people leave the study and, if so, was there something special about them?
D
6. How do the results fit with previous research in the area?
• id the researchers base their work on a thorough literature review?
D
7. What does this research mean for clinical practice?
• Is the study purpose an important clinical issue?
Adapted with permission from Melnyk BM, Fineout-Overholt E, editors. Evidence-based practice in nursing and healthcare:
a guide to best practice [forthcoming]. 2nd ed. Philadelphia: Wolters Kluwer Health/Lippincott Williams and Wilkins.
may inform thinking, such as or a meta-analysis—is the most a
­ ppraisal process (to ­ ppear in
a
d
­ escriptive research, expert opin- reliable and the best evidence to f
­ uture installments of this series).
ions, or guidelines, they put them answer their clinical question. Creating a study evaluation
aside. Carlos explains that they’ll Using a critical appraisal table. Carlos provides an online
be used later to support Rebecca’s guide. Carlos recommends that template for a table where Re-
case for having a rapid response the team use a critical appraisal becca and Chen can put all the
team (RRT) in her hospital, sh­ checklist (see Critical Appraisal data they’ll need for the RCA.
ould the evidence point in that Guide for Quantitative Studies) Here they’ll record each study’s
direction. to help evaluate the 15 studies. essential elements that answer the
After the studies—including This checklist is relevant to all three questions and begin to ap-
those in the “I don’t know” studies and contains questions praise the 15 studies. (To use this
group—are categorized, 15 of about the essential elements of template to create your own eval-
the original 26 remain and will research (such as, pur­ ose of the
p uation table, download the Eval­
be ­ncluded in the RCA: three
i study, sample size, and major uation Table Template at http://
systematic reviews that include variables). links.lww.com/AJN/A10.)
one meta-analysis (Level I evi- The questions in the critical ap­
dence), one randomized con- praisal guide seem a little strange EXTRACTING THE DATA
trolled trial (Level II evidence), to Rebecca and Chen. As they re- Starting with level I evidence
two cohort studies (Level IV evi- view the guide together, Carlos studies and moving down the
dence), one retrospective pre- explains and clarifies each ques- hierarchy list, the EBP team takes
post study with historic controls tion. He suggests that as they try each study and, one by one, finds
(Level VI evidence), four preex- to figure out which are the essen- and enters its essential elements
perimental (pre-post) interven- tial elements of the studies, they into the first five columns of
tion studies (no control group) focus on answering the first three the evaluation table (see Table
(Level VI ­ vidence), and four EBP
e questions: Why was the study 1; to see the entire table with
implementation projects (Level done? What is the sample size? all 15 studies, go to http://links.
VI ­ vidence). Carlos reminds
e Are the instruments of the major lww.com/AJN/A11). The team
Rebecca and Chen that Level I variables valid and reliable? The discusses each element as they
e
­ vidence—a systematic review remaining questions will be ad- enter it, and tries to determine if
of randomized controlled trials dressed later on in the critical it meets the criteria of the critical
ajn@wolterskluwer.com AJN ▼ July 2010 ▼ Vol. 110, No. 7 49

4. 50
Table 1. Evaluation Table, Phase I
First Author (Year) Conceptual Design/Method Sample/Setting Major Variables Studied Measure- Data Findings Appraisal:
Framework (and Their Definitions) ment Analysis Worth to
Practice
Chan PS, et al. None SR N = 18 studies IV: RRT
Arch Intern Med Purpose: effect of RRT on DV1: HMR
2010;170(1):18-26. HMR and CR Setting: acute care hos- DV2: CR
• Searched 5 databases
pitals; 13 adult, 5 peds
AJN ▼ July 2010
from 1950-2008, and
▼
“grey literature” from Average no. beds: NR
MD conferences
• Included only studies
Attrition: NR
with a control group
McGaughey J, et al. None SR (Cochrane review) N = 2 studies IV: RRT
Vol. 110, No. 7
Cochrane Database Purpose: effect of RRT DV1: HMR
Syst Rev 2007;3: on HMR 24 adult hospitals
CD005529. • Searched 6 databases
from 1990-2006 Attrition: NR
• Excluded all but 2
RCTs
Winters BD, et al. None SR N = 8 studies IV: RRT
Crit Care Med Purpose: effect of RRT on DV1: HMR
2007;35(5): HMR and CR Average no. beds: 500 DV2: CR
1238-43. • Searched 3 databases
from 1990-2005 Attrition: NR
• Included only studies
with a control group
Hillman K, et al. None RCT N = 23 hospitals IV: RRT protocol for HMR Note:
Lancet 2005; Purpose: effect of RRT on Average no. beds: 340 6 months CR • Criteria for
365(9477): 2091-7. CR, HMR, and UICUA • Intervention group
• 1 AP
rates of activating
(n = 12) • 1 ICU or ED RN
UICUA RRT
• Control group
DV1: HMR (unexpected
(n = 11) deaths, excluding DNRs)
DV2: CR (excluding
Setting: Australia DNRs)
DV3: UICUA
Attrition: none
Shaded columns indicate where data will be entered in future installments of the series.
AP = attending physician; CR = cardiopulmonary arrest or code rates; DNR = do not resuscitate; DV = dependent variable; ED = emergency department; HMR: hospital-wide mor-
tality rates; ICU = intensive care unit; IV = independent variable; MD = medical doctor; NR = not reported; Peds = pediatric; RCT = randomized controlled trial; RN = registered
nurse; RRT = rapid response team; SR = systematic review; UICUA = unplanned ICU admissions.
ajnonline.com

5. appraisal guide. These elements— suggests they leave the column in. find­ngs, not to compare them
i
such as purpose of the study, sam- He says they can further discuss with other like studies. Rebecca
ple size, and major variables—are this point later on in the process realizes that she enjoys this kind
typical parts of a research report when they synthesize the studies’ of conversation, in which she
and should be presented in a pre­ findings. As Rebecca and Chen and Chen have a voice and can
dictable fashion in every study review each study, they enter its contribute to a deeper under-
so that the reader understands citation in a separate reference list standing of how research impacts
what’s being reported. so that they won’t have to create practice.
As Rebecca and Chen con-
tinue to enter data into the table,
they begin to see similarities and
Usually the important information in a study differences across studies. They
mention this to Carlos, who tells
can be found in the abstract. them they’ve begun the process
of synthesis! Both nurses are en-
couraged by the fact that they’re
learning this new skill.
As the EBP team continues to this list at the end of the pro­­
cess. The MERIT trial is next in the
review the studies and fill in the The reference list will be shared stack of studies and it’s a good
evaluation table, they realize that with colleagues and placed at the trial to use to illustrate this phase
it’s taking about 10 to 15 minutes end of any RRT policy that re- of the RCA process. Set in Aus-
per study to locate and enter the sults from this ­ ndeavor.
e tralia, the MERIT trial1 examined
information. This may be because Carlos spends much of his whether the introduction of an
when they look for a description time answering Rebecca’s and RRT (called a medical emergency
of the sample, for example, it’s Chen’s questions concerning how team or MET in the study) would
important that they note how the to phrase the information they’re reduce the incidence of cardiac
sample was obtained, how many entering in the table. He suggests arrest, unplanned admissions to
patients are included, other char- that they keep it simple and con- the ICU, and death in the hospi-
acteristics of the sample, as well sistent. For example, if a study tals studied. See Table 1 to follow
as any diagnoses or illnesses the indicated that it was implement- along as the EBP team finds and
sample might have that could be ing an RRT and hoped to see a enters the trial data into the table.
important to the study outcome. change in a certain outcome, the Design/Method. After Rebecca
They discuss with Carlos the like- nurses could enter “change in and Chen enter the citation infor-
lihood that they’ll need a few ses- [the outcome] after RRT” as the mation and note the lack of a con­
sions to enter all the data into the purpose of the study. For studies ceptual framework, they’re ready
table. Carlos responds that the examining the effect of an RRT to fill in the “Design/Method”
more studies they do, the less on an outcome, they could say as column. First they enter RCT
time it will take. He also says the purpose, “effect of RRT on for randomized controlled trial,
that it takes less time to find the [the outcome].” Using the same which they find in both the study
information when study reports words to describe the same pur- title and introduction. But MERIT
are clearly written. He adds that pose, even though it may not have is called a “cluster-­ andomised
r
usually the important informa- been stated exactly that way in controlled trial,” and cluster is a
tion can be found in the abstract. the study, can help when they term they haven’t seen before.
Rebecca and Chen ask if it compare studies later on. Carlos explains that it means that
would be all right to take out Rebecca and Chen find it frus- hospitals, not individuals or pa-
the “Conceptual Framework” trating that the study data are tients, were randomly assigned to
column, since none of the stud- not always presented in the same the RRT. He says that the likely
ies they’re reviewing have con- way from study to study. They reason the researchers chose to
ceptual frameworks (which help ask Carlos why the authors or randomly assign hospitals is that
guide researchers as to how a journals wouldn’t present similar if they had randomly assigned
study should proceed). Carlos information in a similar manner. i
­ndividual patients or units, oth-
r
­ eplies that it’s helpful to know Carlos explains that the purpose ers in the hospital might have
that a study has no framework of publishing these studies may heard about the RRT and poten-
underpinning the research and have been to disseminate the tially influenced the outcome.
ajn@wolterskluwer.com AJN ▼ July 2010 ▼ Vol. 110, No. 7 51

6. To randomly assign hospitals the RRTs were activated and pro- continue the work—as long as
(
­ instead of units or patients) to vided their protocol for calling the Carlos is there to help.
the intervention and comparison RRTs. However, these elements In applying these principles
groups is a cleaner research de- might be helpful to the EBP team for evaluating research studies
sign. later on when they make decisions to your own search for the evi-
dence to answer your PICOT
question, ­ emember that this se-
r
ries can’t contain all the available
Keep the data in the table consistent by using infor­ ation about research meth­
m
od­ logy. Fortunately, there are
o
simple, inclusive terminology. many good resources available in
books and online. For example,
to find out more about sample
size, which can affect the likeli-
To keep the study purposes about implementing an RRT in hood that researchers’ results oc­
con­ istent among the studies in
s their hospital. So that they can cur by chance (a random finding)
the RCA, the EBP team uses inclu- come back to this information, rather than that the intervention
sive terminology they developed they place it in the last column, brought about the expected out-
after they noticed that different “Appraisal: Worth to Practice.” come, search the Web using terms
trials had different ways of de- After reviewing the studies to that describe what you want to
scribing the same objectives. Now make sure they’ve captured the know. If you type sample size
they write that the purpose of the essential elements in the evalua- findings by chance in a search en-
MERIT trial is to see if an RRT tion table, Rebecca and Chen still gine, you’ll find several Web sites
can reduce CR, for cardiopulmo- feel unsure about whether the in- that can help you better under-
nary arrest or code rates, HMR, formation is complete. Carlos stand this study essential.
for hospital-wide mortality rates, r
­ eminds them that a system-wide Be sure to join the EBP team
and UICUA for unplanned ICU practice change—such as the in the next installment of the se-
admissions. They use those same change Rebecca is exploring, that ries, “Critical Appraisal of the
terms consistently throughout the of implementing an RRT in her Evi­ ence: Part II,” when Rebecca
d
evaluation table. hospital—requires careful consid- and Chen will use the MERIT
Sample/Setting. A total of 23 eration of the evidence and this is trial to illustrate the next steps
hospitals in Australia with an only the first step. He cautions in the RCA process, complete
average of 340 beds per hospi- them not to worry too much the rest of the evaluation table,
tal is the study sample. Twelve about perfection and to put their and dig a little deeper into the
hospitals had an RRT (the inter- efforts into understanding the studies in order to detect the
vention group) and 11 hospitals i
­nformation in the studies. He re- “keepers.” ▼
didn’t (the control group). minds them that as they move on
Major Variables Studied. The to the next steps in the critical Ellen Fineout-Overholt is clinical profes­
sor and director of the Center for the
independent variable is the vari- appraisal process, and learn even Advancement of Evidence-Based Practice
able that influences the outcome more about the studies and proj- at Arizona State University in Phoenix,
(in this trial, it’s an RRT for six ects, they can refine any data in where Bernadette Mazurek Melnyk
is dean and distinguished foundation
months). The dependent vari­ the table. Rebecca and Chen feel professor of nursing, Susan B. Stillwell
able is the outcome (in this case, uncomfortable with this uncer- is clinical associate professor and pro­
HMR, CR, and UICUA). In this tainty but decide to trust the pro- gram coordinator of the Nurse Educator
Evidence-Based Practice Mentorship
trial, the outcomes didn’t include cess. They continue extracting Program, and Kathleen M. Williamson
do-not-resuscitate data. The RRT data and entering it into the table is associate director of the Center for the
was made up of an attending phy­ even though they may not com- Advancement of Evidence-Based Practice.
Contact author: Ellen Fineout-Overholt,
sician and an ICU or ED nurse. pletely understand what they’re ellen.fineout-overholt@asu.edu.
While the MERIT trial seems entering at present. They both
to perfectly answer Rebecca’s r
­ ealize that this will be a learn- REFERENCE
PICOT question, it contains ele- ing opportunity and, though the 1. Hillman K, et al. Introduction of
ments that aren’t entirely relevant, le­ rning curve may be steep at
a the medical emergency team (MET)
system: a cluster-randomised con­
such as the fact that the research- times, they value the outcome of trolled trial. Lancet 2005;365(9477):
ers collected information on how improving patient care enough to 2091-7.
52 AJN ▼ July 2010 ▼ Vol. 110, No. 7 ajnonline.com