Research Methods in Education 6th Edition

1. META-ANALYSIS, RESEARCH
SYNTHESES AND SYSTEMATIC
REVIEWS
© LOUIS COHEN, LAWRENCE
MANION & KEITH MORRISON

2. STRUCTURE OF THE CHAPTER
• Evidence-based research
• Meta-analysis
• Research syntheses and systematic reviews

3. EVIDENCE-BASED RESEARCH
• Comes from randomized controlled trials and
research syntheses
• Based on accumulated evidence of ‘what works’
• Evidence-based approaches are necessary in order
to:
– challenge the imposition of unproven practices
– solve problems and avoid harmful procedures
– create improvement that leads to more effective learning
• Looks at effect size, not only statistical significance

4. META-ANALYSIS
• The analysis of other analyses
• Uses data from randomized controlled trials
• Aggregates and combines the results of
comparable studies into a coherent account to
discover main effects
• Often uses statistical processes
• Looks at effect size, not only statistical
significance
• Combines the results of small-scale studies
• Uses transparent means to draw conclusions

5. CONCERNS ABOUT META-ANALYSIS
• Lacks comprehensiveness, as it makes selective use of
subsets of studies.
• Misrepresents research findings.
• Relies too heavily on significance tests.
• Understates effect sizes as a way of supporting hypotheses.
• Fails to recognize that sampling error can play a part in
creating variations in findings.
• Overlooks differing/conflicting research findings.
• Published research is favoured over unpublished research.
• Interaction effects overlooked in favour of main effects.
• Unclear how it differentiates between ‘good’ and ‘bad’
research, e.g. between rigorous and poorly constructed
research.

6. CONCERNS ABOUT META-ANALYSIS
• Difficult to draw logical conclusions from studies that
use different interventions, measurements,
definitions of variables, and participants.
• Results from poorly designed studies take their place
alongside results from higher quality studies.
• Multiple results from a single study are used, making
the overall meta-analysis appear more reliable than
it is, since the results are not independent.
• Dangerous consequences because its apparent
objectivity and precision may disguise procedural
invalidity in the studies.
• Risks trying to synthesize studies which are
insufficiently similar to each other to permit this.

7. CONCERNS ABOUT META-ANALYSIS
• Fails to examine critically the evidence,
methods and conclusions of previous reviews
• Overlooks the extent to which findings from
research are mediated by sample
characteristics
• Overlooks the importance of intervening
variables in research
• Unreplicable because procedures for
integrating the research findings not made
explicit

8. AN EIGHT-STEP PROCESS FOR
META-ANALYSIS
Step 1: Identify the variables for focus (dependent and
independent).
Step 2: Identify all the studies which feature the
variables in which the researcher is interested.
Step 3: Code each study for those characteristics that
might be predictors of outcomes and effect sizes (e.g.
age, gender, ethnicity).
Step 4: Estimate the effect sizes for each pair of
variables (dependent and independent variable),
weighting the effect-size by the sample size.

9. AN EIGHT-STEP PROCESS FOR
META-ANALYSIS
Step 5: Calculate the mean and the standard
deviation of effect-sizes across the studies, i.e. the
variance across the studies.
Step 6: Determine the effects of sampling errors,
measurement errors and range of restriction.
Step 7: If a large proportion of the variance is
attributable to the issues in Step 6, then the average
effect-size may be an accurate estimate of
relationships between variables.
Step 8: If a large proportion of the variance is not
attributable to the issues in Step 6, review those
characteristics of interest which correlate with the
study effects.

10. EFFECT SIZE
• The effect size is calculated as:
groupcontroltheofdeviationStandard
group)controlofMeangroupalexperimentof(Mean −

11. RESEARCH SYNTHESES AND
SYSTEMATIC REVIEWS
• Include studies that are not solely randomized
controlled trials.
• Apply explicit protocols and rigorous criteria for
searching for and selecting relevant primary, usually
empirical studies, and to remove bias.
• Combine qualitative and quantitative studies.
• Follow a standard set of stages.
• Are accountable, able to be replicated and updated.
• Must be relevant and useful to users.
• Must answer specific research questions.
• Are evidence-based.

12. RESEARCH SYNTHESES AND
SYSTEMATIC REVIEWS
• Clear, transparent criteria for:
– inclusion and exclusion of studies
– standards for acceptable methodological rigour
– relevance to the topic in question
– scope of the studies included
– team approaches to reviewing in order to reduce
bias
– adoption of a consistent and clearly stated
approach to combining information from across
different studies
– careful, relevant conclusions and recommendations

13. SEQUENCE FOR CONDUCTING
SYSTEMATIC REVIEWS
• Formulate the problem and kinds of relevant research
evidence to answer the research question or
hypothesis.
• Examine the research designs in the studies used, the
treatment of the main effects, the kind of research, the
conceptual relevance of the studies, i.e. screen the
studies.
• Search the literature.
• Gather information from studies (including developing
and using a coding guide, identifying predictor and
outcome variables, research designs used, sampling,
context, statistics used, effect sizes).

14. SEQUENCE FOR CONDUCTING
SYSTEMATIC REVIEWS
• Evaluate the quality of studies (decide which studies
to include and exclude, identify problems in the
research studies used, evaluate the suitability of the
research design for the research synthesis, evaluate
the quality and rigour of the research).
• Analyze and integrate the outcomes of studies.
• Interpret the evidence.
• Present the results.