1. 1
Mr. Fida Karim
Market Research Analyst
By Fida Karim Hunzai- Research
Analyst

2.  High quality evaluation research uses the
scientific method to investigate the effectiveness
of programs and practices.
 Some evaluation studies are higher quality than
others, and the research consumer must learn to
distinguish among them.
 One important index of quality is the rigor of the
study’s research design.
2
By Fida Karim Hunzai- Research
Analyst

3.  An evaluation’s research design is its structure.
 At its most stingy, the structure consists of
 the new program
 study participants
 timing of outcome measures
 The research consumer must be able to
distinguish between experimental and
observational designs and what makes them
internally and externally valid.
3
By Fida Karim Hunzai- Research
Analyst

4.  An RCT is an experimental study in which eligible individuals or
groups of individuals (e.g., schools, communities) are assigned at
random to receive one of several programs or interventions.

 The group in an experiment that receives the specified program
is called the experimental group.
 The term control group refers to another group assigned to the
experiment, but not for the purpose of being exposed to the
program.
 The performance of the control group usually serves as a standard
against which to measure the effect of the program on the
experimental group.
4
By Fida Karim Hunzai- Research
Analyst

5.  The control program may be typical practice (“usual care”), an
alternative practice, or a placebo (a treatment or program
believed to be inert or innocuous).
 Random assignment means that people end up in the
experimental or in the control group by chance rather than by
choice.
 In some randomized studies, the participants and investigators do
not know which participants are in the experimental or the control
groups: This is the double-blind experiment. When participants do
not know, but investigators do, this is called the blinded trial.
5
By Fida Karim Hunzai- Research
Analyst

6.  Randomized controlled trials are sometimes
called true experiments because, at their best,
they can demonstrate causality.
 That means that, in theory, the researcher can
assume that if participants in an RCT achieve
desirable outcomes, the program caused them.
6
By Fida Karim Hunzai- Research
Analyst

7. Two commonly used randomized control designs
are:
1. Concurrent controls in which two (or more)
groups are randomly constituted and they are
studied at the same time (concurrently).
2. Wait-list controls in which one group receives
the program first; if the program appears to be
effective, participants on the wait list receive it.
 Participants are randomly assigned to the experimental and
wait-list groups.
7
By Fida Karim Hunzai- Research
Analyst

8.  True experiments (randomized controlled trials) are contrasted
with non randomized controlled trials and observational studies.
 In non randomized controlled trials, the control group is
predetermined (without random assignment) to be comparable to
the program group
 Non randomized controlled trials are also called quasi
experiments.
 In observational designs, the evaluator does not intervene but
studies the effects of already existing programs
 Observational designs are sometimes referred to as descriptive.
8
By Fida Karim Hunzai- Research
Analyst

9.  Non randomized controlled trials rely on
participants who --
1) volunteer to join the study OR
2) are geographically close to the study site OR
3) conveniently turn up (at a clinic, school) while the study
is being conducted
 Because the study groups are opportunistically
rather than randomly composed, study group
characteristics (age, sex) may not be balanced
before (at baseline) the study begins.
 Baseline differences between groups may
confound the study’s results.
9
By Fida Karim Hunzai-
Research Analyst

10. 10
 Typical confounding variables include age, educational level,
motivation, severity of illness, social structure, and income.
 Evaluation researchers worry that study groups in non
randomized trials will differ from one another at baseline, and
the study’s findings will be compromised.
 They aim to create study groups that are as similar to one
another as possible (equivalent) at baseline or before
“treatment.”
 Among the strategies commonly used to ensure equivalence is
one called matching.
By Fida Karim Hunzai- Research
Analyst

11.  Matching requires selecting pairs of participants or
clusters of individuals who are comparable to one
another on important variables.
 A researcher who is interested in comparing the acuity of
vision among smokers and non smokers can try to
balance the two groups by selecting pairs of smokers and
non smokers who are same age, sex, and have the same
medical history
 Statistical methods such as analysis of covariance
and propensity score analysis are sometimes used
to deal with the problem of confounding after the
data are collected for the study.
11
By Fida Karim Hunzai- Research
Analyst

12.  Time-series Designs
 Time-series designs are longitudinal studies that enable the
researcher to monitor change from one time to the next.
 They are sometimes called repeated measures analyses.
 Interrupted or Single Time-series
 The interrupted or single time-series design without a control
group involves repeated measurement of a variable (e.g.,
reported crime) before and after implementation of a program.
 The goal is to evaluate whether the program has "interrupted"
or changed a pattern established before the program's
implementation.
12
By Fida Karim Hunzai- Research
Analyst

13.  Self-Controlled or Pretest-Post Test Designs
 Each participant is measured on some important program
variable and serves as his or her own control.
 Participants are usually measured twice (at baseline and after
program participation), but they many be measured multiple
times afterward as well.
 Historical Controls
 Investigators compare outcomes among participants who
receive a new program with outcomes among a previous group
of participants who received the standard program.
13
By Fida Karim Hunzai- Research
Analyst

14. Cohort Designs
 A cohort is a group of people who have something in common
and who remain part of a study group over an extended period of
time.
 In public health research, cohort studies are used to describe and
predict the risk factors for a disease and the disease's cause,
incidence, natural history and prognosis. They tend to be
extremely large studies.
 Cohort studies may be prospective or retrospective. With a
prospective design, the direction of inquiry is forward in time,
while with a retrospective design, the direction is backward in
time.
14
By Fida Karim Hunzai- Research
Analyst

15.  Case Control Designs
 Case-control designs are generally retrospective.
They are used to explain why a phenomenon
currently exists by comparing the histories of two
different groups, one of which is involved in the
phenomenon.
 For example, a case control design might be used to
help understand the social, demographic, and
attitudinal variables that distinguish people who at the
present time have been identified with frequent
headaches from those who do not at the present time
have frequent headaches.
15
By Fida Karim Hunzai- Research
Analyst

16.  Cross-Sectional Designs
 Cross-sectional designs result in a portrait of one or
many groups at one period of time.
 They are sometimes called descriptive, pre
experimental, or survey designs.
16
By Fida Karim Hunzai- Research
Analyst

17.  Internal Validity is the ability to make accurate
inferences about a program’s outcomes and
effectiveness. (Program A caused Outcome A.)
 External validity refers to the extent to which
the results are applicable to other programs,
populations, and settings. Another term for
external validity is generalizability.
17
By Fida Karim Hunzai- Research
Analyst

18.  The research consumer wants to be sure that
Program A which was conducted in Setting A
1. caused Outcome A (internal validity) and
2. will be effective in Settings B and C, etc. (external
validity)
Study design flaws can “threaten” internal and external
validity.
18
By Fida Karim Hunzai- Research
Analyst

19. Threat to Validity Explanation
Selection Group characteristics (e.g., age,
gender) are not evenly balanced
History Unanticipated outside events
occur while study is in progress
Maturation Participants change or mature
Testing Taking one test effects
performance on the second
19
By Fida Karim Hunzai- Research
Analyst

20. Threat to Validity Explanation
Instrumentation Changes occur in measures or
observers
Statistical regression Participants who are selected
because of extremely high or
low scores return to the average
over time
Attrition Participants drop out of the
study in a non random manner
Expectancy Participants or researchers
have expectations of the
program and study activities
20
By Fida Karim Hunzai- Research
Analyst

21. Threat to External Validity Explanation
Interaction between selection
and experiment
A unique mixture is created that
is unlikely to occur again
Reactive effects of testing Completing the baseline
measures influences program
participation uniquely
Reactive effects of
experimentation (Hawthorne
Effect)
Study participants behave
uncharacteristically because
they are being observed
Multiple program interference Complementary activities may
influence participants’ behavior
and study outcomes
21Dated: 09202013
By Fida Karim Hunzai - Research
Analyst