SFU SIAT Comprehensive Examination

Comprehensive Examination
Research Methods Question Presentation
Vitomir Kovanovic
School of Interactive Arts and Technology,
Simon Fraser University
Surrey, BC Canada
vitomir kovanovic@sfu.ca
December 9, 2013

Research Methods Question
Learning Analytics is a ﬁeld that draws on numerous data about learners and the
context in which learning happens.
Studying learning in the quasi-experimental setting brings certain issues that have
to be considered. Identify those issues; show how those drive choice of data
analytics and techniques, type of outcomes and issues related to validity of
ﬁndings. Illustrate on the case study of your choice.
Vitomir Kovanovic (SIAT) Comprehensive Examination December 9, 2013 2 / 50

Agenda
1 Research Plan
Research Plan Overview
Theoretical Foundation
Proposed Approach
2 Quasi-Experimental Research in Learning Analytics
Overview of Learning Analytics Research Methods
Quasi-Experimental Research
Analytical Approaches
3 Conclusions

Research Plan
Research Plan
1 Research Plan
Proposed Approach
3 Conclusions

Research Plan Research Plan Overview
High Level Overview
Problem
How we can best use the vast amount of data to improve learning process in
context of the socially-enabled learning environments?
Solution
Provide instructors with information on student’s learning so that appropriate
instructional interventions can be planned and implemented.
Provide learners with the feedback on their learning progress so that they can
reﬂect more objectively on their own learning.
Approach
Adopt techniques of Learning Analytics to provide relevant information about
students’ learning.

Research Plan Theoretical Foundation
Theoretical Foundations: Community of Inquiry - CoI
Community of Inquiry
Community of Inquiry is a conceptual framework outlying important constructs
that deﬁne worthwhile educational experience in distance education setting.
Three presences:
• Social presence: relationships and social
climate in a community.
• Cognitive presence: phases of cognitive
engagement and knowledge construction.
• Teaching presence: instructional role
during social learning.
CoI model is:
• Extensively researched and validated
• Adopts Content Analysis for assessment of
presences

Research Plan Theoretical Foundation
Theoretical Foundations: Community of Inquiry - CoI
Issues and challenges:
• Used for analysis of learning long after courses are over.
• Require substantial manual and time consuming work for content analysis of
discussion messages for assessment of the levels of three presences.
• Not explaining reasons behind observed levels of presences.
• Not providing suggestions or guidelines for instructors to direct their
pedagogical decisions.

Research Plan Proposed Approach
Learning Analytics
Learning Analytics
Measurement, collection, analysis and reporting of data about learners and their
contexts, for purposes of understanding and optimizing learning and the
environments in which it occurs.
Process of Learning Analytics
Five approaches of Social Learning Analytics
(Ferguson and Shum, 2012):
• Social Network Analytics
• Content Analytics
• Discourse Analytics
• Disposition Analytics
• Context Analytics

Research Plan Proposed Approach
Learning Analytics for Communities of Inquiry
Automatic
Content Analysis
LMS Database
System-use
Feature Extractor
Prediction Component
SNA Extraction
and
Metrics Calculation
NER of
Learning Concepts
Interventions and
Feedback
Student Profiling

Quasi-Experimental Research in Learning Analytics
Quasi-Experimental Research in Learning Analytics
1 Research Plan
Proposed Approach
3 Conclusions

Quasi-Experimental Research in Learning Analytics Overview of Learning Analytics Research Methods
High-level View of Educational Research Methods
Most common classiﬁcation:
• Quantitative research
• Scientiﬁc Method
• Positivism
• Systematic investigation
• Use of statistics
• Qualitative research
• Postpositivism
• Social constructionism
• Study of human behavior
• Frequent use of narratives &
interviews
• Mixed research
• Multiple perspectives
• Best of both worlds
Based on purpose of research:
• Basic Research
• Broadening the knowledge
• Applied Research
• Solve Practical Problems
Additional types of research:
• Design-based research
• Naturalistic study of interventions
• Iterative
• Real-world setting
• Researchers & Practitioners
collaboration
• Action Research
• Practical approach to inquiry
• Practitioners lead research
• Improving practice

High-level View of Educational Research Methods
Most common classiﬁcation:
• Quantitative research
• Scientiﬁc Method
• Positivism
• Systematic investigation
• Use of statistics
• Qualitative research
• Postpositivism
• Social constructionism
• Study of human behavior
• Frequent use of narratives &
interviews
• Mixed research
• Multiple perspectives
• Best of both worlds
Based on purpose of research:
• Basic Research
• Broadening the knowledge
• Applied Research
• Solve Practical Problems
Additional types of research:
• Design-based research
• Naturalistic study of interventions
• Iterative
• Researchers & Practitioners
collaboration
• Action Research
• Practical approach to inquiry
• Practitioners lead research
• Improving practice
Choice of research method is NOT a personal preference choice!
It should depend on the study goals and research questions.

Quantitative Research Methods
Four basic types of quantitative research in education (Creswell, 2012):
• Descriptive (survey) research.
• What are the characteristics of a population?
• Frequencies, averages
• Often based on surveys
• Correlational research
• Synonymous with nonexperimental research
• Simply observes the size and direction of a relationship among variables
• Experimental research
• Intervention is deliberately introduced to observe its eﬀects.
• Random assignment to diﬀerent conditions
• “Gold standard” for research.
• Quasi-experimental research
• Almost experimental
• Assignment to conditions is not random

Experimental Research
”The only legitimate way to try to establish a causal connection
statistically is through the use of randomized experiments.”
(Utts, 2005)
• Direct manipulation of
independent variable
• Random assignment to
diﬀerent treatments
• Usually at laboratory
• Can be very expensive
• Not always possible or desirable
• Issues of external and construct
validity

Why Random Assignment?

Quasi-Experimental Research in Learning Analytics Quasi-Experimental Research
Quasi-Experimental (QE) Research
“by deﬁnition, quasi-experiments lack random assignment.”
(Shadish, Campbell, and Cook, 2010)
Two types of assignment:
• Self-assignment
• Administrative assignment
Two types of Quasi-Experiments:
• “Person-by-treatment” experiments
• In laboratories
• At least one variable measured,
one manipulated
• “Natural” experiments
• No control over assignment
Types of QE (Fife-Schaw, 2006):
• One-group design
• Non-equivalent control groups design
• Posttest only
• Pretest and posttest
• Interrupted time-series designs
• Single
• Multiple

One-group Design
Most common:
• Pretest-posttest design: O X O
Variations:
• Posttest-only design: X O
• Double pretest design: O O X O
• Nonequivalent dependent variable design: [Oa Ob] X [Oa Ob]
• Removed-treatment design (ABA): O X O ¬X O
• Repeated-treatment design (ABAB): O X O ¬X O X O

Non-equivalent Control Groups Design
Most common:
• Pretest-posttest design: Intervention Group: O X O
Control Group: O O
Variations:
• Posttest-only design: Intervention Group: X O
Control Group: O
• Double pretest design: Intervention Group: O O X O
Control Group: O O O
• Switching replications: Intervention Group: O X O
Control Group: O O X O

Interrupted Time-Series Designs

Multiple Time-Series Designs

Advantages and Disadvantages of Quasi-experiments
Advantages over true experiments:
• Often easier to set up
• Often higher external validity
• Less ethical considerations
• Can sometimes give even better
results
Disadvantages of quasi-experiments:
• Confounding
• Issues with internal validity
• Some ethical issues still exist
• Less control
• Sometimes comparable groups do
not exist
• Replicability problem

Internal Validity
Deﬁnition
The extent to which a research design allows us to infer that a relationship
between two variables is a causal one or that the absence of a relationship
indicates the lack of causal relationship (Cramer and Howitt, 2004)
True Experimenal
Research
Quasi-Experimental
Research
Correlational
Research
High Internal Validity
Low External Validity
Low Internal Validity
High External Validity
Quantitative research validity

Internal Validity Treats
• Confounding
• Selection bias
• Experimenter bias
• History
• Contamination (Diffusion)
• Maturation effects
• Testing effect
• Regression toward the mean
• Instrumentality
• Mortality
• Competition

Quasi-Experimental Research in Learning Analytics Analytical Approaches
Analytical Approaches for Resolving Validity Threats
• Double pretest
• Nonequivalent dependent variable
• Regression discontinuity
• Matching

• Double pretest
• Matching

Double Pretest
Main idea
Use the “dry run” in order to check for internal validity treats
• Maturation eﬀect
• Regression toward the mean
• Testing eﬀect

• Double pretest
• Matching

Nonequivalent Dependent Variable
Hypothetical example
CBC starts running a 26-week long TV program called “Learning
Alphabet” where every week children learn one new letter.
How eﬀective is it for helping children to learn alphabet?

Naive approach (Pretest-posttest design): O X O
1 Select a sample of children
2 Assess their knowledge of the alphabet
3 Run the TV show for 26 weeks
4 Assess their knowledge again
• If the diﬀerence is positive, then this is the evidence
that “Learning Alphabet” helped children to learn alphabet

Typical approach (Posttest-only NEGD design): Intervention Group: X O
Control Group: O
2 Assess their knowledge of the alphabet
4 Split the sample into two groups
based on whether they watched TV show or not
• If the diﬀerence is bigger in the group that watched the show,
then this is the evidence that “Learning Alphabet” helped children
to learn alphabet

Alternative approach: [Oa Ob] X [Oa Ob]:
2 Assess their knowledge of the first 13 letters of the alphabet
3 Assess their knowledge of the second 13 letters of the alphabet
• If the difference is bigger for first 13 letters than for the second 13 letters,
then this is the evidence that “Learning Alphabet” helped children to learn
alphabet

Main idea
Use of the 2nd
variable which is very related, but should not be
aﬀected by the intervention

• Double pretest
• Matching

Regression Discontinuity
Real-world Example
How is the Gates Millennium Scholars (GMS) Program
related to college students time use and activities?
(DesJardins et al., 2010)
Administered by Bill & Melinda Gates Foundation, provides $1 billion in
scholarships over 20 year period.
• Provide help to high achieving, low-income students
• Provides a scholarship that covers unmet ﬁnancial need
• Selection criteria:
• Cognitive: 3.3 high school GPA
• Non-cognitive: must be over a given threshold for a his ethnic group

Regression Discontinuity idea
Students just above and below the cutoﬀ: distributed in an approximately random
fashion, similar to randomized trial

Results of the DesJardins et al. (2010) study:
• Receiving a scholarship significantly reduces hours of work
• No significant effects on hours spent studying, relaxing, or in extracurricular
activities.
• Receiving a scholarship significantly lowers hours worked by African
Americans and Asian Americans.
• Receiving a scholarship significantly does not lower hours worked by Latino
Americans.
• Latino Americans report significantly higher levels of participation in cultural
events relative to other racial/ethnic groups.

• Double pretest
• Matching

Matching
Real-world example
What are the eﬀects of coaching on SAT scores?
(Domingue and Briggs, 2009; Rock and Powers, 1998)
Rock and Powers (1998) study:
• ’95 Survey data about whether students used coaching
• Large section of the report dedicated to “who seeks coaching” question

Matching
Overall Idea of Matching
Balance treated and control groups on observable characteristics
as much as possible.

Matching
Use matching when:
• Very few subjects from both groups are directly comparable
(e.g., regression to the mean)
• Selection of control subjects is hard due to the high dimensionality
Typical procedure:
• Run logistic regression on all available variables to retroactively predict
chance of receiving a treatment.
• This chance is called “propensity score”
• Create control group where instances have similar propensity scores.
Two common approaches:
• “Optimal” matching
• Subclassification
Matching vs OLS Regression:
• In matching only untreated units that are similar are used as control group!
• OLS Regression can overestimate the effect of treatment due to the large
differences between groups (e.g., selection bias)

Matching

Matching
Domingue and Briggs (2009) study
• Used propensity score matching to answer similar question as the study by
Rock and Powers (1998)
• Data from 2002 Education Longitudinal Survey
• Contains many variables about high school students:
• Their SAT scores
• Many demographic variables
• Data about whether they used coaching to prepare for SAT

Matching
Domingue and Briggs (2009) study results

Arnold and Pistilli (2012) Example
Study by Arnold and Pistilli (2012) presented at Learning Analytics and
Knowledge ’12 conference.
Typical example of LA system evaluation studies.
Authors presented Course Signals (CS) system:
• Feedback to students on their progress
• Some of the courses use this system
• Eﬀect of use of Course Signals on student retention
• Comparing students who used CS and who didn’t

Arnold and Pistilli (2012) Results

Arnold and Pistilli (2012) Example
Challenges:
• Weak proof of causality
• Selection bias
• History eﬀect
• Students who used CS have lower SAT scores
• “this aspect needs to be further investigated” (Arnold and Pistilli, 2012)
Ways to improve:
• Make treatment and control groups more comparable
• Matching

Conclusions
Conclusions
1 Research Plan
Proposed Approach
3 Conclusions

Conclusions
Conclusions
Current Trends:
• Many of shown methods are not commonly used
• Learning Analytics research is most often correlational
• Big promise of Learning Analytics
• Still to deﬁne its “standard of research”
• Bigger impact of LA on educational practice

Conclusions
Conclusions
How this relates to my own research?
• Learning Analytics data are mostly observational
• Mostly working with “medium” data and will probably analyze BIG data for
my PhD research
• Currently working on quasi-experimental data sets
• Overall idea is to advance both theoretical knowledge in CoI area and
practical impact of it by development of LA systems
• Making stronger claims about causality

References I
Arnold, Kimberly E. and Matthew D. Pistilli (2012). “Course Signals at
Purdue: Using Learning Analytics to Increase Student Success”. In:
Proceedings of the 2Nd International Conference on Learning Analytics and
Knowledge, 267270. doi: 10.1145/2330601.2330666.
http://doi.acm.org/10.1145/2330601.2330666.
Cramer, Duncan and Dennis Howitt (2004). The SAGE Dictionary of
Statistics: A Practical Resource for Students in the Social Sciences. English.
Sage Pub.
Creswell, John W (2012). Educational research: planning, conducting, and
evaluating quantitative and qualitative research. English. Pearson.
DesJardins, Stephen L. et al. (2010). “A Quasi-Experimental Investigation of
How the Gates Millennium Scholars Program Is Related to College Students
Time Use and Activities”. en. In: Educational Evaluation and Policy
Analysis 32.4, pp. 456–475. doi: 10.3102/0162373710380739.
http://epa.sagepub.com/content/32/4/456.

References II
Domingue, Ben and Derek C Briggs (2009). “Using linear regression and
propensity score matching to estimate the effect of coaching on the SAT”.
In: Multiple Linear Regression Viewpoints 35.1, pp. 12–29.
Ferguson, Rebecca and Simon Buckingham Shum (2012). “Social learning
analytics: five approaches”. In: Proceedings of 2nd International Conference
on Learning Analytics & Knowledge, p. 23.
Fife-Schaw, Chris (2006). “Quasi-experimental Designs”. en. In: Research
Methods in Psychology.
Rock, Donald A. and Donald E. Powers (1998). Effects of Coaching on SAT
I: Reasoning Scores. Tech. rep. 98-6. The College Board.
http://research.collegeboard.org/publications/content/2012/
05/effects-coaching-sat-i-reasoning-scores.
Shadish, William R, Donald Thomas Campbell, and Thomas D Cook
(2010). Experimental and quasi-experimental designs for generalized causal
inference. English. Wadsworth, Cengage Learning.
Utts, Jessica M (2005). Seeing through statistics. English. Thomson,
Brooks/Cole.

SFU SIAT Comprehensive Examination

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (12)

Ähnlich wie SFU SIAT Comprehensive Examination

Ähnlich wie SFU SIAT Comprehensive Examination (20)

Mehr von Vitomir Kovanovic

Mehr von Vitomir Kovanovic (14)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

SFU SIAT Comprehensive Examination