1. Types of research studies
1
CASE STUDIES
EXPERIMENTS
DESIGN STUDIES
SURVEYS
MITE 6025 Session 2 (van Aalst)9/26/2013
2. What is an experiment?
A study that tests a hypothesis about
a causal effect using at least two
independent samples
The experimental sample receives the
intervention of interest
The control sample does not receive
that intervention but is equivalent in all
other respects
2
2.Experiments
MITE 6025 Session 2 (van Aalst)9/26/2013
3. Hypotheses
A hypothesis is an educated guess
about what is going to happen during the
experiment
It usually is based on an implication
(prediction) of a theory
MITE 6025 Session 2 (van Aalst) 39/26/2013
2.Experiments
4. Pre- and post-test
Having experimental and control
samples not enough
Also need to test both samples before
(pre-test) and after the treatment
(post-test)
4 MITE 6025 Session 2 (van Aalst)9/26/2013
2.Experiments
5. Simplest experimental design
Experimental Control
• Pretest on
dependent
variable
• Pretest on
other relevant
variables
• Pretest on
dependent
variable
• Pretest on
other relevant
variables
• Treatment • No treatment
• Posttest on
dependent
variable
• Posttest on
dependent
variable
Generalization to
population
Random selection
from population
Random
assignment to
conditions
MITE 6025 Session 2 (van Aalst) 59/26/2013
2.Experiments
6. Sample results for dependent
variable
Experimental Control
Pretest 50.1 (10.2) 48.4 (10.7)
Posttest 72.0 (8.5) 60.9 (10.9)
The samples are
similar at the
pretest
The experimental
sample
outperforms the
control sample on
post-test
Need to calculate
the probability of
getting this effect
MITE 6025 Session 2 (van Aalst) 69/26/2013
2.Experiments
7. Preparation (pretests)
English proficiency test
Chinese proficiency test
General science achievement test
(Chinese)
Specific science test (same language as
task)
MITE 6025 Session 2 (van Aalst) 79/26/2013
2.Experiments
8. Task design
A reading task in English and Chinese
Exactly the same content in both versions
The same reading level in each language
Not too long
MITE 6025 Session 2 (van Aalst) 89/26/2013
2.Experiments
9. Finishing up (post-tests)
Specific science post-test for both
samples
Time for completion for both samples
MITE 6025 Session 2 (van Aalst) 99/26/2013
2.Experiments
10. Summary
Stage Experimental sample Control sample
Preparation • English proficiency
test
• Chinese proficiency
test
• General science test
(Chinese)
• Specific science test
(English)
• English proficiency
test
• Chinese proficiency
test
• General science test
(Chinese)
• Specific science test
(Chinese)
intervention • Reading task
(English)
• Task completion
time
• Reading task
(Chinese)
• Task completion
time
Finishing up • Specific science test
(English)
• Specific science test
(Chinese)
MITE 6025 Session 2 (van Aalst) 109/26/2013
2.Experiments
11. Sample results
Experimental (n = 30) Control (n = 30)
English test 672 (130) 520 (160)
Chinese test 620 (150) 590 (140)
General science test 50.1 (10.0) 42.8 (10.3)
Specific science pre-test 40.0 (9.5) 35.8 (10.2)
Specific science
posttest
60.1 (9.9) 57.9 (13.2)
Completion time 17.5 (3.5) min 25.7 (3.9) min
MITE 6025 Session 2 (van Aalst) 119/26/2013
2.Experiments
12. Extensions
More than one treatment
More dependent variables
More control variables
MITE 6025 Session 2 (van Aalst) 129/26/2013
2.Experiments
13. Quasi-experiments
When used
random selection and/or random assignment
are impossible
Sample equivalence is difficult to achieve
E.g. the use of in-tact classes
Assignment to classes not random
One class does the treatment, the other class
does not
Classes may be taught by different teachers
MITE 6025 Session 2 (van Aalst) 139/26/2013
2.Experiments