Overview of the PhD research of Marcel Schmitz as presented at the doctoral consortium of ECTEL in 2017

1. Opportunities and
Challenges in using
Learning Analytics in
Learning Design
PhD Student: Marcel Schmitz1
Supervisors: Evelien van Limbeek1, Roger
Bemelmans1, Hendrik Drachsler2,3,4
1 Zuyd University of Applied Sciences, NL
2 Open University, NL
3 Goethe University Frankfurt, DE
4 German Institute for International Educational Research (DIPF),
DE

2. Current
situation
2
Design
Time
Run
Time
Learning
Design
Behavior
during
Learning
Action
after
analysis

3. LA4LD
Solution
- “Info in Run Time”
- (Metacognitive)
competences
3
Design
Time
Run
Time
Learning
Design
Behavior
during
Learning
Action
after
analysis
Actions
after
analysis
Metacognitive
Competences
Analysis
Learning
Analytics
Dashboards

4. Main research
question
Can Learning Analytics, by use of Learning Dashboards,
improve the Learning Design of a course during run-time
and in that way make the learning process more efficient,
effective or increase the satisfaction of teachers and
students?
4

5. Sub research
questions
1. What (learning) data can be made accessible (at Zuyd University)?
2. Which analyses of the educational data are supportive to give students and
teachers more insights in the learning
progress/process?
3. What metacognitive competences do students and teachers need to interpret
and act appropriately on the LA dashboard?
4. What is the additional value of using LA in run-time on study effectiveness,
efficiency and satisfaction?
5. How can metacognitive competences be stimulated in a LA for LD solution?
6. Which interventions will be done by teachers on a course during run-time and
after the run-time (design-time) of a course when using Learning Dashboards?
7. How does the use of Learning dashboard affect student learning during run-time
of a course?
5

6. Design based
research
Context:
- Zuyd bachelor
ICT, students and
teachers
6
Substudy 1:
• Task 1:State of the art related to LA4LD
solution (evidence)
• Task 2: Identification context and users
(practice)

7. Substudy 1
Questions: 1,2 and 3
State of the art and identification of context and users.
• Task 1: State of the art (previous research) applied to our
proposal/solution: Learning Design, Learning Analytics, Learning
Dashboards, Metacognitive competences*
• Task 2: Experiment with LA Dashboard (SURF project at Zuyd)
• Task 3: Relating data of multiple Dutch HEI’s on the LA Dashboard-
experiment (SURF) to study results and MSLQ*-scores
• Task 4: Focusgroups/Creative sessions/Surveys with students and
teachers at Zuyd on LA4LD
7
*MSLQ: Motivational Strategies for Learning Questionnaire (a validated survey for assessment of
motivational strategies is learners such as Metacognitive competences (Pintrich and DeGroot, 1990)

8. Substudy 2
Question: 4
Developing a Learning Dashboard for Learning Design with
run-time usage
• Task 1: Based on substudy 1 preliminary design of a
LA4LD solution with students and teachers at Zuyd
• Task 2: Pretest/pilottest of LA4LD solution in existing ICT-
course (iteration 1)
• Task 3: Evaluation/monitoring by surveys and focus
groups and EFLA* with end users
8
*Evaluation Framework for Learning Analytics (Scheffel & Drachsler, 2016) – Maren Scheffel will present on
this instrument during ECTEL-conference

9. Substudy 3
Question: 5
Addressing metacognitive competences in a LA for LD
solution.
• Task 1: Based on substudy 2 an analysis is made
metacognitive competences needed to effectively use the
LA4LD solution (and validated with students and
teachers)
• Task 2: Solution redesign in collaboration with teachers
and students of Zuyd ICT incorporating results of Task 1
• Task 3: Pretest/pilottest of LA4LD solution in existing ICT-
course (iteration 2)
• Task 4: Evaluation/monitoring by surveys and focus
groups and EFLA* with end users
9

10. Substudy 4
Questions: 6 and 7
Evaluation of the Learning Dashboard and insight in
interventions and behavioral change
• Task 1: Solution redesign in collaboration with teachers
and students of Zuyd ICT incorporating results of
substudy 2 and 3
• Task 2: Implementation of LA4LD solution in existing ICT-
course (iteration 3)
• Task 3: Evaluation/monitoring by surveys and focus
groups and EFLA with end users
• Task 4: Additional surveys, thinking aloud sessions,
interviews, observations on how student and teacher
behavior has changed by the LA4LD solution, final
analyses on MSQL-scores
10

11. marcel.schmitz@zuyd.nl
@marcelschmitz