1.
Concept-Level Design Analytics for Blended Courses
Laia Albó1, Jordan Barria-Pineda2, Peter Brusilovsky2 and Davinia Hernández-Leo1
1TIDE, Interactive & Distributed Technologies for Education,
Universitat Pompeu Fabra, Barcelona (Spain)
2PAWS, Personalized Adaptive Web Systems Lab,
University of Pittsburgh, Pennsylvania (US)
ECTEL Conference September 18th, 2019

2.
Analytics layers for learning design (AL4L) framework
2
Community Analytics
metrics and patterns of design activity
Design Analytics
metrics of design decisions and related aspects that
characterize learning designs
Learning Analytics
metrics of learners’ engagement,
achievement... aligned with design intent
Hernández-Leo, D., Martinez-Maldonado, R., Pardo, A., Muñoz-Cristóbal, J. A., & Rodríguez-Triana, M. J. Analytics for
learning design: A layered framework and tools, British Journal of Educational Technology.

3.
Smart learning Content (SLC)
3
● Lack of work in supporting educators in making data-informed design
decisions when designing a blended course and planning learning activities.
● Smart Learning Content (SLC)*: Interactive activities, which engages students in
exploration and provides real-time performance feedback.
Our paper explores some approaches for fine-grained design analytics
focused on visualizing critical metadata associated with SLC
* Brusilovsky, P. et.al. Increasing Adoption of Smart Learning Content for Computer Science Education. Working Group Reports
of the 2014 on Innovation and Technology in Computer Science Education Conference, Uppsala, Sweden, ACM (2014).

4.
Type of SLC Comprehension Construction
Examples
WebEx Anim_Ex PCEX
Problems
Quizjet PCEX_ch PCRS
Smart learning Content (SLC)
4
• SLC for a HE computer science JAVA course.
• Each SLC activity is associated with different programming concepts.

5.
edCrumble: https://ilde2.upf.edu/edcrumble/
5

6.
Concept-Level Design Analytics for Blended Learning
6
● 3 types of visualizations

7.
Concept-Level Design Analytics for Blended Learning
7
● Using the concept-level design analytics integrated into edCrumble*
● Video: http://tiny.cc/edCrumbleECTEL
* Albó, L., & Hernández-Leo, D. (2018). edCrumble: designing for learning with data analytics. In V. Pammer-Schindler, M. Pérez-
Sanagustín, H. Drachsler, R. Elferink, & M. Scheffel (Eds.), Lifelong Technology-Enhanced Learning. EC-TEL 2018. Lecture Notes in
Computer Science, vol 11082. (pp. 605–608). Leeds, UK: Springer, Cham.

8.
Within-subjects User Study
Baseline interface vs. visualizations interface
8

9.
Methodology
9
● Task 1: Design a Lab session for Lecture 4 using 8 (problems) activities in total.
a) Covering key concepts introduced during the class.
b) Balancing between comprehension and program construction.
● Task 2: Design a Practice session for the Lecture 4 using 20 (examples & problems)
activities in total.
a) Covering key concepts introduced during the class.
b) Balancing between examples and problems.
c) Balancing between comprehension and program construction.

10.
Methodology: within-subjects user study
10
Participants: 10 domain experts (six female) qualified as introductory
programming instructors.

11.
Methodology: within-subjects user study
11
Data collection: action logs, learning design outcomes, 3 questionnaires

12.
Results
12

13.
Results: Cognitive load
13
NASA_TLX questionnaire (adjusted ratings considering weights)

14.
Results: Action analysis
14
User actions with the system while performing each task during the two
treatments.

15.
Results: Learning Design outcomes
15
Learning designs’ outcomes. *(p=0.011; p<0.05) T-test between subjects.

16.
Results: Learning Design outcomes
16
Mean of the number of times that a concept is practiced in the activities selected.
● 13 previous concepts, 8 current concepts, and a counter for future concepts.

17.
Results: Learning Design outcomes
17
Mean of the number of times that a concept is practiced in the activities selected.
● Comparison between the types of Smart Learning Content

18.
Results: User Feedback Analysis
18
All 10 instructors stated that:
● they preferred to use the interface with the visualization.
● visualizations allowed them to more effectively design their sessions.
The visualizations were easy to understand and were useful in deciding which
activity to choose.
● six out of ten found the ‘Type of session’ comparison to be more useful.
All three comparisons were meaningful for the instructors in order to create their
course designs.

19.
Conclusions
19
The use of concept-level design analytics:
● may reduce the cognitive load of design tasks (MD!)
● has facilitated the selection of the most suitable activities (=time)
● has allowed the almost complete disappearance of future concepts
● has changed the behavior of instructors in the process of selecting the
activities (previewing contribution vs. looking deeper into their content)
The differences in concept-level balance between the conditions were small.
● Higher impact in selecting just few activities.

20.
Future work
20
Explore and evaluate the use of concept-level design analytics:
● with a larger sample,
● in other educational contexts,
● comparing different types of visualizations.
Connection between design analytics and learning analytics extracted from the
Smart Learning Content.

21.
Thank you
Tool: https://ilde2.upf.edu/edcrumble/
Demo video: http://tiny.cc/edCrumbleECTEL
Contact: laia.albo@upf.edu
@LaiaAlbo
Research collaboration supported by
the SEBAP Research Mobility Award
https://www.amicsdelpais.com/
Research partially funded by:
NSF DRL 1740775
“la Caixa Foundation”(CoT project, 100010434)
FEDER, Grant Numbers:
MDM-2015-0502, TIN2014-53199-C3-3-R,
TIN2017-85179-C3-3-R.