Sven Charleer defended his PhD thesis on designing and evaluating student-facing learning dashboards. Over the course of his research, he developed 7 dashboards across 3 learning settings involving over 100 students and 20 instructors. His work resulted in 19 publications and has been cited over 120 times. Going forward, Charleer aims to conduct longer term evaluations and deployments of dashboards at additional universities.

1. Sven Charleer
Designing and Evaluating
Student-facing Learning Dashboards
Lessons Learnt
Public PhD Defence
July 6, 2017
Augment
Lab

2. 2
https://www.apple.com2

3. 3
3

4. LEARNING ANALYTICS
4
“The measurement, collection, analysis, and reporting of data about learners
and their contexts, for purpose of understanding and optimising learning and
the environments in which it occurs”
J. L. Santos. Learning Analytics and Learning Dashboards: a Human- Computer Interaction Perspective. PhD dissertation, KU Leuven, 2015.
G. Siemens. “Learning analytics: envisioning a research discipline and a domain of practice”. Proceedings of the 2nd
International Conference on Learning Analytics and Knowledge . ACM. 2012, pp. 4–8.
session
course
degree
year
Microlevel
intro

5. AUTOMATE AUGMENT
Engelbart, D. C. (2001). Augmenting human intellect: a
conceptual framework (1962). PACKER, Randall and
JORDAN, Ken. Multimedia. From Wagner to Virtual Reality.
New York: WW Norton & Company, 64-90.
VS
5
Arnold, Kimberly E., and Matthew D. Pistilli. Course Signals at Purdue:
Using learning analytics to increase student success. Proceedings of
the 2nd International Conference on Learning Analytics and Knowledge.
ACM, 2012.

6. LEARNING DASHBOARDS
6
“A Learning Dashboard is a single display that aggregates different indicators
about learner(s), learning process(es) and/or learning context(s) into one or
multiple visualisations.”
B. A. Schwendimann, M. J. Rodríguez-Triana, A. Vozniuk, L. P. Prieto, M. S. Boroujeni, A. Holzer, D. Gillet, and P. Dillenbourg. Understanding learning at a glance: An overview of learning
dashboard studies. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge, pages 532–533. ACM, 2016.
K. Verbert, E. Duval, J. Klerkx, S. Govaerts, and J. L. Santos. Learning Analytics Dashboard Applications. American Behavioral Scientist, 57(10):1500–1509, 2013.
Perceived benefits
Design guidelines
intro
awareness
(self) reflection
sense making
impact
data
questions
answers
behavior

7. 7
DESIGN-BASED RESEARCHintro
External Input through collaboration with participants:
Teachers
Students
Experts

8. 8
DESIGN-BASED RESEARCHintro
Multiple dependent variables

9. 9
DESIGN-BASED RESEARCHintro
Rapid prototyping

10. 10
CONTEXTintro
ch2
ch3
ch4

11. Santos, J.L., Verbert, K., Govaerts, S., Duval, E.: Addressing learner issues with stepup!: an evaluation. In:
Proceedings of the Third International Conference on Learning Analytics and Knowledge, ACM (2013) 14–22
CREATING EFFECTIVE LEARNING DASHBOARDSch2
11

12. PATH OF EFFORT
OUTCOMES
INFLUENCES
12 https://www.flickr.com/photos/lockechrisj/
LEARNER PATHSch2

13. RQ1: How should we visualise learner data to support students to
explore the path from effort to outcomes?
RQ2: How can we promote students, inside and outside the
classroom, to actively explore this effort to outcomes path?
CREATING EFFECTIVE LEARNING DASHBOARDSch2
13

14. 14
EVALUATIONSch2

15. TwitterBlogs
15
EVALUATIONSch2

16. Inquiry-Based Learning16
EVALUATIONSch2

17. Abstract the LA data
Provide access to the artefacts
Augment the abstracted data
Provide access to teacher and peer feedback
17
RESULTS
RQ1: What are relevant learning traces, and how should we visualise these
data to support students to explore the path from effort to outcomes?
ch2

18. 18
RESULTS
RQ2: How can we promote students, inside and outside the classroom, to
actively explore this effort to outcomes path?
Visualise the learner path
Integrate LA into the workflow
Facilitate collaborative exploration of the LA data
ch2

19. 19
ABSTRACT / AUGMENTch2

20. 20
INTEGRATE / AUGMENTch2

21. 21
FEEDBACK / ARTEFACTSch2
21

22. 22
VISUALISE LEARNER PATHch2

23. 23
Geoff’s hypothesis
Geoff’s data collection
Geoff’s conclusion
Angela’s data collection
Angela activities Geoff’s activities
VISUALISE LEARNER PATHch2

24. 24
Geoff’s hypothesis
Geoff’s data collection
Geoff’s conclusion
Angela’s data collection
Angela activities Geoff’s activities
VISUALISE LEARNER PATHch2

25. 25
Geoff’s hypothesis
Geoff’s data collection
Geoff’s conclusion
Angela’s data collection
Angela activities Geoff’s activities
VISUALISE LEARNER PATHch2

26. 26
Geoff’s hypothesis
Geoff’s data collection
Geoff’s conclusion
Angela’s data collection
Angela activities Geoff’s activities
VISUALISE LEARNER PATHch2

27. 27
Geoff’s hypothesis
Geoff’s data collection
Geoff’s conclusion
Angela’s data collection
Angela activities Geoff’s activities
VISUALISE LEARNER PATHch2

28. 28
Geoff’s hypothesis
Geoff’s data collection
Geoff’s conclusion
Angela’s data collection
Angela activities Geoff’s activities
VISUALISE LEARNER PATHch2

29. 29
Geoff’s hypothesis
Geoff’s data collection
Geoff’s conclusion
Angela’s data collection
Angela activities Geoff’s activities
VISUALISE LEARNER PATHch2

30. 30
Geoff’s hypothesis
Geoff’s data collection
Geoff’s conclusion
Angela’s data collection
Angela activities Geoff’s activities
VISUALISE LEARNER PATHch2

31. 31
COLLABORATIONch2

32. 32
CONTRIBUTIONSch2
Guidelines published at European Conference on Technology Enhanced Learning
(25% acceptance)
S. Charleer, J. Klerkx, E. Duval, T. De Laet, and K. Verbert. Creating effective learning analytics dashboards: Lessons learnt.
Adaptive and Adaptable Learning: 11th European Conference on Technology Enhanced Learning, EC-TEL 2016, Lyon,
France, September 13-16, 2016, Proceedings, pages 42–56, Cham, 2016. Springer International Publishing
14 papers
collaboration with
Stanford University - University of Technology Sidney - University of the Basque
Country - Murdoch University Perth - Curtin University Perth
Program Committees
EC-TEL, LAK, Cross-LAK, ARTEL, EC-TEL Doctoral Consortium
European Commission for the weSPOT project

33. 33
CONTRIBUTIONSch2
Austria, Bulgaria, Netherlands, Slovenia, United Kingdom
33

34. 34
CONTRIBUTIONSch2
34

35. BALANCED DISCUSSION IN THE CLASSROOMch3
over- and under-participation
35

36. BALANCED DISCUSSION IN THE CLASSROOMch3
RQ3: What are the design challenges for ambient Learning
Dashboards to promote balanced group participation in
classrooms, and how can they be met?
RQ4: Are ambient Learning Dashboards effective means for
creating balanced group participation in classroom settings?
36

37. EVALUATION SETUPch3
case study 1
# participants 12 students
deployment
1 3h session with dashboard
1 3h session without dashboard
evaluation
class discussion
questionnaires (perceived distraction/
awareness/usefulness)
activity/quality logging
case study 2
# participants 19 students
deployment
half 3h session without dashboard
half 3h session with dashboard
evaluation
questionnaires (perceived importance
feedback/motivation)
activity/quality logging
37

38. Visualise balance in an abstract and neutral way
Add the qualitative dimension to the visualisation
Create a realistic picture of the classroom situation
38
RESULTS
RQ3: What are the design challenges for ambient LDs to promote balanced
group participation in classrooms, and how can they be met?
ch3

39. Ambient dashboards as support for teacher/presenter
Ambient dashboards raise awareness of the invisible
Ambient feedback information can activate students
39
RESULTS
RQ4: Are ambient LDs effective means for creating balanced group
participation in classroom settings?
ch3

40. 40
ABSTRACT / NEUTRALch3
Group 1 Group 2
Group 5 Group 3
Group 4

41. CONVERGENCEch3
-0.2
0
0.2
0.4
0.6
0.8
%distancefromaverage
-0.2
0
0.2
0.4
0.6
0.8
0 10.5
% of time of feedback session passed
%ofdistancefromaverage
without visualisation
with visualisation
group 1
group 2
group 3
group 4
group 5
group 6
41

42. 42
CONTRIBUTIONS
Published Special Issue on Awareness and Reflection in
Technology-Enhanced Learning, IJTEL
(8/21 submissions accepted)
Charleer, S., Klerkx, J., Duval, E., De Laet, T. and Verbert, K. (2017) ‘Towards balanced
discussions in the classroom using ambient information visualisations’, Int. J. Technology Enhanced
Learning, Vol. 9, Nos. 2/3, pp.227–253.
Basis for new research collaboration
with the University of Sidney
ch3

43. SUPPORTING ADVISER-STUDENT DIALOGUEch4
43
lack of data-based feedback

44. SUPPORTING ADVISER-STUDENT DIALOGUEch4
RQ5: What are the design challenges for creating a Learning
Dashboard to support study advice sessions, and how can they be
met?
RQ6: How does such a Learning Dashboard contribute to the role
of the adviser, student, and dialogue?
44

45. EVALUATION SETUPch4
design
# participants
17 study advisers (preliminary feedback)
5 study advisers (iterative feedback)
approach
brainstorms/observations
iterative design
evaluation
# participants 5 study advisors
deployment
Engineering Science, Engineering Science:
Architecture
97 sessions (15-30min per session)
evaluation
15 sessions observed
questionnaires perceived usefulness
45

46. Data Confidence
Collaboration
Adviser’s role
46
RESULTS
RQ6: How does such a Learning Dashboard contribute to the role of the
adviser, student, and dialogue?
ch4
RQ5: What are the design challenges for creating a Learning Dashboard to
support study advice sessions, and how can they be met?
Authorship
Visual Encoding
Ethics

47. RESULTSch4
47

48. RESULTSch4
48

49. RESULTSch4
S. Claes, N. Wouters, K. Slegers, and A. V. Moere. Controlling In-the-Wild Evaluation Studies of Public Displays. pages 81–84, 2015.
49

50. 50
CONTRIBUTIONSch4
Published in IEEE Transactions on Learning Technologies
11% acceptance rate, IF 2.267
S. Charleer, A. Vande Moere, J. Klerkx, K. Verbert, and T. De Laet. Learning analytics dashboards
to support adviser-student dialogue. IEEE Transaction on Learning Technologies, 18 pages
Deployed at
Engineering Science, Engineering Science: Architecture, Maths,
Biology, Physics, Geology, Geography, Biochemistry, Informatics,
Bio-engineering, Engineering Technology (3 campuses)
15 study advisers during 165 sessions

51. BEYOND THE TEXTxtra
Assistent Bachelor/Master Engineering Science courses
Advisor Master Theses
3 European Projects
51

52. 52
PROBLEEMOPLOSSEN & ONTWERPEN 3xtra
52

53. 53
PROBLEEMOPLOSSEN & ONTWERPEN 3xtra
53

54. MASTER THESESxtra
54

55. 55
MASTER THESESxtra
55

56. Discover, interact Personal Space
(Mendeley)
V. Alvarez, J. Klerkx, S. Charleer, E. Duval, and D. Moor. Science 2.0 and visual data exploration using augmented reality. In Proceedings of the Workshop on Collaboration Meets
Interactive Surfaces: Walls, Tables, Tablets and Phones (CMIS), ACM International Conference on Interactive Tabletops and Surfaces (ITS), Dresden, D, October 2014, pages 1–6,
Oct. 2014
MASTER THESESxtra
56

57. 57
EUROPEAN PROJECTSxtra
57

58. S. Charleer, J. Klerkx, E. Duval, T. De Laet, and K. Verbert. Faceted Search on Coordinated Tablets and Tabletop: a Comparison. In Proceedings of the 8th
ACM SIGCHI Symposium on Engineering Interactive Computing Systems, Engineering Interactive Computing Systems, Brussels, Belgium, 21-24 June 2016,
pages 165–170. ACM, June 2016
FACETED SEARCH TABLETS vs TABLETOPxtra
58

59. 59
S. Charleer, J. Klerkx, and E. Duval. PoPI: Glyph Designs for Collaborative Filtering on Interactive Tabletops. In E. Bertini, J. Kennedy, and E. Puppo, editors, Eurographics Conference on
Visualization (EuroVis) - Short Papers. The Eurographics Association, 2015
GLYPHS FOR COLLABORATIONxtra
59

60. 60
SUMMARYwrp
3 learning settings
7 dashboards
100+ students, 20 instructors, 17 study advisers
19 publications

61. 61
SUMMARYwrp

62. 62
Citation indices Since 2012
# citations 120
h-index 7
i10-index 4
CITATIONSwrp
62

63. 63
FUTURE (ONGOING) WORKwrp
Ground work for long-term evaluations/deployments
Leiden University
Student union requests faculty deployment
KU Leuven has shown interest in dashboard university-wide

64. ȡ
leerkracht callcenter verkoper administratie management
actionable
non-actionable
Opleiding: MS Excel
Opleiding: MS PowerPoint
studies
8.9%
7.2%
Engels
Frans
Duits
talen
5.1%
2.4%
Antwerpen
Brussel
regio
3.9%
2.2%
leeftijd dagen werkloos
0 20
18 65 0 800
0 10 0 10
FUTURE (ONGOING) WORKwrp
64

65. 65
FUTURE (ONGOING) WORKwrp
65

66. 66
66

67. Designing and Evaluating
Student-facing Learning Dashboards
Lessons Learnt
Sven Charleer
Public PhD Defence
July 6, 2017
(╯°□°)╯︵ ┻━┻