1. Experimental Research
Chapter Thirteen

2. Experimental Research
• Chapter Thirteen

3. Uniqueness of Experimental Research
• Experimental Research is unique in two important
respects:
1) Only type of research that attempts to influence a
particular variable
2) Best type of research for testing hypotheses about cause-
and-effect relationships
• Experimental Research looks at the following
variables:
• Independent variable (treatment)
• Dependent variable (outcome)

4. Major Characteristics of
Experimental Research
• The researcher manipulates the independent
variable.
• They decide the nature and the extent of the
treatment.
• After the treatment has been administered,
researchers observe or measure the groups
receiving the treatments to see if they differ.
• Experimental research enables researchers
to go beyond description and prediction, and
attempt to determine what caused effects.

5. Essential Characteristics of
Experimental Research
Comparison of Groups:
• The experimental group receives a treatment of
some sort while the control group receives no
treatment.
• Enables the researcher to determine whether the
treatment has had an effect or whether one
treatment is more effective than another.
Manipulation of the Independent Variable:
• The researcher deliberately and directly determines
what forms the independent variable will take and
which group will get which form.

6. Essential Characteristics of
Experimental Research
Randomization
• Random assignment is similar but not identical to random
selection.
• Random assignment means that every individual who is
participating in the experiment has an equal chance of
being assigned to any of the experimental or control
groups.
• Random selection means that every member of a
population has an equal chance of being selected to be a
member of the sample.
• Three things occur with random assignments of subjects:
1) It takes place before the experiment begins
2) Process of assigning the groups takes place
3) Groups should be equivalent

7. Control of Extraneous Variables
• The researcher has the ability to control many
aspects of an experiment.
• It is the responsibility of the researcher to control for
possible threats to internal validity.
• This is done by ensuring that all subject
characteristics that might affect the study are
controlled.

8. Most Common Ways to
Eliminate Threats
• Randomization
• Hold certain variables constant
• Build the variable into the design
• Matching
• Use subjects as their own control
• Analysis of Covariance (ANCOVA)

9. Poor Experimental Designs
• The following designs are considered weak since they do
not have built-in controls for threats to internal validity
– The One-Shot Case Study
• A single group is exposed to a treatment and its effects are assessed
– The One-Group-Pretest-Posttest Design
• Single group is measured both before and after a treatment exposure
– The Static-Group Comparison Design
• Two intact groups receive two different treatments

10. The One-Shot Case Study
 A single measure is recorded after the
treatment in administered.
 Study lacks any comparison or control of
extraneous influences.
 To remedy this design, a comparison could
be made with another group.
 Diagrammed as:

11. The One-Group Pretest-Posttest
Design
 Subjects are measured before and after
treatment is administered.
 Uncontrolled-for threats to internal validity
exist.
 To remedy this design, a comparison group
could be added.
 Diagrammed as:

12. The Static-Group Comparison Design
 Use of 2 existing, or intact groups.
 Experimental group is measured after being
exposed to treatment.
 Control group is measured without having
been exposed to the treatment.
 Diagrammed as:

13. The Static-Group Pretest-Posttest
Design
 Pretest is given to both groups.
 “Gain” or “change” = pretest score - posttest
score.
 Better control of subject characteristics
threat.
 A pretest raises the possibility of a testing
threat.

14. True Experimental Designs
• The essential ingredient of a true experiment is random
assignment of subjects to treatment groups
• Random assignments is a powerful tool for controlling
threats to internal validity
– The Randomized Posttest-only Control Group Design
• Both groups receiving different treatments
– The Randomized Pretest-Posttest Control Group Design
• Pretest is included in this design
– The Randomized Solomon Four-Group Design
• Four groups used, with two pre-tested and two not pre-tested

15. The Randomized Posttest-Only
Control Group Design
 Experimental group tested after treatment
exposure.
 Control group tested at the same time
without exposure to experimental
treatment.
 Includes random assignment to groups.
 Threats to internal validity –
mortality, attitudinal, implementation, data
collector bias, location and history.

16. Example of a Randomized Posttest-
Only Control Group Design

17. The Randomized Pretest-Posttest
Control Group Design
 Experimental group tested before and after
treatment exposure.
 Control group tested at same two times
without exposure to experimental
treatment.
 Includes random assignment to groups.
 Pretest raises the possibility of a pretest
treatment interaction threat.

18. Example of a Randomized Pretest-
Posttest Control Group Design

19. The Randomized Solomon Four-
Group Design
 Combines pretest-posttest with control
group design and the posttest-only with
control group design.
 Provides means of controlling the interactive
test effect and other sources of extraneous
variation.
 Does include random assignment.
 Weakness: requires a large sample.

20. Example of a Randomized Solomon
Four-Group Design

21. Random Assignment with Matching
• To increase the likelihood that groups of
subjects will be equivalent, pairs of subjects
may be matched on certain variables.
• Members of matched groups are then
assigned to experimental or control groups.
• Matching can be mechanical or statistical.

22. Mechanical and Statistical Matching
• Mechanical matching is a process of pairing two
persons whose scores on a particular variable are
similar.
• Statistical matching does not necessitate a loss of
subjects, nor does it limit the number of matching
variables.
– Each subject is given a “predicted” score on the dependent
variable, based on the correlation between the dependent
variable and the variable on which the subjects are being
matched.
– The difference between the predicted and actual scores
for each individual is then used to compare experimental
and control groups.

23. A Randomized Posttest-Only Control
Group Design

24. Quasi-Experimental Designs
• Do not include the use of random assignments
but use other techniques to control for threats
to internal validity:
– The Matching-Only Design
• Similar except that no random assignment occurs
– Counterbalanced Design
• All groups are exposed to all treatments but in a different order
– Time-Series Design
• Involves repeated measures over time, both before and after
treatment

25. The Matching-Only Design
 Random assignment is not used.
 An alternative to random assignment of
subjects but never a substitute for random
assignment.

26. Counterbalanced Designs
 Each group is exposed to all treatments but in a
different order.
 The effectiveness of the various treatment can
be determined by comparing the average score
for all groups on the posttest for each
treatment.
 e.g. Results (Means) from a Study Using a
Counterbalanced Design.

27. Time-Series Design
 Involves periodic measurements on the
dependent variable for a group of test units.
 After multiple measurements, experimental
treatment is administered (or occurs naturally).
 After the treatment, periodic measurements are
continued in order to determine the treatment
effect.
 The threats to internal validity –
history, instrumentation, and testing.
 Infrequently used due to extensive amount of
data collection.

28. Possible Outcome Patterns in a
Time-Series Design

29. Factorial Designs
• Factorial Designs extend the number of relationships that may
be examined in an experimental study.
• They are modifications of either the posttest-only control
group or pretest-posttest control group designs which permit
the investigation of additional independent variables.
• They also allow a researcher to study the interaction of an
independent variable with one or more other variables
(moderator variable).

30. Using a Factorial Design to Study
Effects of Method and Class Size on
Achievement

31. Illustration of Interaction and No
Interaction in a 2 by 2 Factorial Design

32. Example of a 4 by 2 Factorial
Design

33. Controlling Threats to Internal Validity
• Subject Characteristics • Testing
• Mortality • History
• Location • Maturation
• Instrument decay • Attitudinal
• Data Collector Characteristics • Regression
• Data Collector bias • Implementation
The above must be controlled to reduce threats to internal validity

34. Effectiveness of Experimental Designs in Controlling Threats to Internal Validity
KEY: (++) = strong control, threat unlikely to occur; (+) = some control, threat may possibly occur;
(–) = weak control, threat likely to occur; (?) = can’t determine; (NA) = threat does not apply
Subject Instru- Data Collec-
Charac- Morta- Loca- ment tor Charac- Data Col- Matur- Atti- Regres- Implemen-
Design teristics lity tion Decay teristics lector Bias Testing History ation tudinal sion tation
One-shot case
study – – – (NA) – – (NA) – – – – –
One group pre-
posttest – ? – – – – – – – – – –
Static group
comparison – – – + – – + ? + – – –
Randomized post-
test-only control
group ++ + – + – – ++ + ++ – ++ –
Randomized pre-
post-test control
group ++ + – + – – + + ++ – ++ –
Solomon four-
group ++ ++ – + – – ++ + ++ – ++ –
Randomized
posttest only
control group
with matched
subjects ++ + – + – – ++ + ++ – ++ –
Matching-only
pre-posttest
control group + + – + – – + + + – + –
Counterbalanced ++ ++ – + – – – ++ ++ ++ ++ –
Time-series ++ – + _ – – – – + – ++ –
Factorial with
randomization ++ ++ – ++ – – + + ++ – ++ –
Factorial without
randomization ? ? – ++ – – + + + – ? –

35. Evaluating the Likelihood of a
Threat to Internal Validity
 Procedures in assessing the likelihood of a threat
to internal validity –
Step 1: Ask: What specific factors either are known to
affect the dependent variable or may logically be
expected to affect this variable?
Step 2: Ask: What is the likelihood of the comparison
groups differing on each of these factors?
Step 3: Evaluate the threats on the basis of how likely
they are to have an effect, and plan to control for
them.

36. Guidelines for Handling Internal
Validity in Comparison Group Studies