Lecture for Master students given at KMI podium, see podcast here http://stadium.open.ac.uk/podium/

1. Potentials and Limitations of
Educational Datasets
Hendrik Drachsler
Open University of the Netherlands

2. Hendrik Drachsler
• Assistant professor at the Centre for Learning
Sciences and Technologies (CELSTEC)
• Track record in TEL projects such as
TENCompetence, SC4L, LTfLL, Handover, dataTEL.
• Main research focus:
– Personalization of learning with information
retrieval technologies, recommender systems and
educational datasets
– Visualization of educational data, data mash-up
environments, supporting context-awareness by
data mining
– Social and ethical implications of data mining in
education
• Leader of the dataTEL Theme Team of the
STELLAR network of excellence (join the SIG on
TELeurope.eu)
• Just recently: new alterEGO project granted by the
Netherlands Laboratory for Lifelong Learning (on
limitations of learning analytics in formal and
informal learning)

3. dataTEL
Potentials and Limitations of Educational Datasets
24.07.2011 MUP/PLE lecture series, Knowledge Media Institute, Open University UK
Hendrik Drachsler #dataTEL
Centre for Learning Sciences and Technology
@ Open University of the Netherlands3

4. Goals of the lecture
1.Motivation or dataTEL
2.The dataTEL project
3.Potentials of dataTEL
4.Open issues of dataTEL
4

5. TEL RecSys Research
5

6. Survey on TEL Recommender
6

7. Survey on TEL Recommender
Manouselis, N., Drachsler, H., Vuorikari, R., Hummel, H. G. K., & Koper, R. (2011).
Recommender Systems in Technology Enhanced Learning. In P. B. Kantor, F.
Ricci, L. Rokach, & B. Shapira (Eds.), Recommender Systems Handbook (pp.
387-415). Berlin: Springer. 6

8. Survey on TEL Recommender
Observation:
Half of the systems (11/20) still at design or prototyping stage
only 8 systems evaluated through trials with human users.
Manouselis, N., Drachsler, H., Vuorikari, R., Hummel, H. G. K., & Koper, R. (2011).
Recommender Systems in Technology Enhanced Learning. In P. B. Kantor, F.
Ricci, L. Rokach, & B. Shapira (Eds.), Recommender Systems Handbook (pp.
387-415). Berlin: Springer. 6

9. Survey on TEL Recommender
Observation:
Conclusion:
Small-scale experiments with a fewdesign or that rate some
Half of the systems (11/20) still at learners prototyping stage
resources only addsevaluated through trialsa knowledge base
only 8 systems little contributions to with human users.
on recommender systems and personalization in TEL.
Manouselis, N., Drachsler, H., Vuorikari, R., Hummel, H. G. K., & Koper, R. (2011).
Recommender Systems in Technology Enhanced Learning. In P. B. Kantor, F.
Ricci, L. Rokach, & B. Shapira (Eds.), Recommender Systems Handbook (pp.
387-415). Berlin: Springer. 6

10. The TEL recommender research is a
bit like this...
7

11. The TEL recommender research is a
bit like this...
We need to design for each domain an
appropriate recommender system that fits the goals, tasks,
and particular constraints
7

12. But...
“The performance results
of different research
efforts in TEL
recommender systems
are hardly comparable.”
(Manouselis et al., 2010)
Kaptain Kobold
http://www.flickr.com/photos/
kaptainkobold/3203311346/
8

13. But...
“The performance results
The TEL recommender
of different research
experiments lack
efforts in TEL
transparency. They need
recommender systems
to be repeatable to test:
are hardly comparable.”
• Validity
(Manouselis et al., 2010)
• Verification
• Compare results Kaptain Kobold
http://www.flickr.com/photos/
kaptainkobold/3203311346/
8

14. How others compare their
recommenders
9

15. How others compare their
recommenders
Although the TEL domain stores plenty of
data everyday in e-learning environments
(LMS, PLEs) there is a lack of shareable
and publicly available datasets.
9

16. Goals of the lecture
1.Motivation or dataTEL
2.The dataTEL project
3.Potentials of dataTEL
4.Open issues of dataTEL
10

17. Who is dataTEL ?
dataTEL is a Theme Team funded by the
STELLAR network of excellence
Riina Stephanie Katrien Nikos Martin Hendrik
Vuorikari Lindstaedt Verbert Manouselis Wolpers Drachsler
11

18. Who is dataTEL ?
dataTEL is a Theme Team funded by the
STELLAR network of excellence
Riina Stephanie Katrien Nikos Martin Hendrik
Vuorikari Lindstaedt Verbert Manouselis Wolpers Drachsler
MAVSEL CEN PT
Social Data
Miguel Joris
Angel Sicillia Klerkx11

19. Who is dataTEL ?
dataTEL is a Theme Team funded by the
STELLAR network of excellence
Riina Stephanie Katrien Nikos Martin Hendrik
Vuorikari Lindstaedt Verbert Manouselis Wolpers Drachsler
MAVSEL CEN PT
Social Data
Miguel Joris
Angel Sicillia Klerkx11

20. dataTEL::Objectives
Make the research on TEL RecSys more comparable by
lowering the entrance barriers for other researchers and
increase the quality.
The required benchmarks therefore are:
1.A collection of public available datasets ranging from
formal to non-formal learning settings
2.An overview of the research results of certain RecSys
technologies on different datasets
3.A common approach to evaluate RecSys in the domain
of TEL
12

21. dataTEL::Objectives
1.Collecting publicly available datasets
2.Sharing policy to (re)use and share datasets
3.Define dataset standards (documentation, pre-
processing)
4.Address privacy and legal protection rights
5.Create evaluation criteria for TEL recommender
systems
6.Create a body of knowledge on personalization in TEL
13

22. dataTEL::Collection
14

23. dataTEL::Collection
15

24. dataTEL::Collection
Drachsler, H., Bogers, T., Vuorikari, R., Verbert, K., Duval, E., Manouselis, N.,
Beham, G., Lindstaedt, S., Stern, H., Friedrich, M., & Wolpers, M. (2010). Issues
and Considerations regarding Sharable Data Sets for Recommender
Systems in Technology Enhanced Learning. Presentation at the 1st Workshop
Recommnder Systems in Technology Enhanced Learning (RecSysTEL) in conjunction
with 5th European Conference on Technology Enhanced Learning (EC-TEL 2010):
Sustaining TEL: From Innovation to Learning and Practice. September, 28, 2010,
Barcelona, Spain. 15

25. dataTEL::Collection
•Collected data is very different with
respect to amount of users and
resources
•Most of the data is very sparse
•Privacy regulations harm data sharing
•Mostly data from R., Verbert, K., Duval, E., Manouselis, N.,
Drachsler, H., Bogers, T., Vuorikari, informal learning
settings
Beham, G., Lindstaedt, S., Stern, H., Friedrich, M., & Wolpers, M. (2010). Issues
and Considerations regarding Sharable Data Sets for Recommender
Systems in Technology Enhanced Learning. Presentation at the 1st Workshop
Recommnder Systems in Technology Enhanced Learning (RecSysTEL) in conjunction
with 5th European Conference on Technology Enhanced Learning (EC-TEL 2010):
Sustaining TEL: From Innovation to Learning and Practice. September, 28, 2010,
Barcelona, Spain. 15

26. dataTEL::Collection
16

27. dataTEL::Collection
16

28. dataTEL::Collection
16

29. dataTEL::Collection
16

30. dataTEL::Body of knowledge
Verbert, K., Drachsler, H., Manouselis, N., Wolpers, M., Vuorikari, R., Beham, G., Duval, E.,
(2011). Dataset-driven Research for Improving Recommender Systems for Learning. Learning
Analytics & Knowledge: February 27-March 1,17 2011, Banff, Alberta, Canada

31. dataTEL::Body of knowledge
Outcomes:
Tanimoto similarity +
item-based CF was
the most accurate.
Verbert, K., Drachsler, H., Manouselis, N., Wolpers, M., Vuorikari, R., Beham, G., Duval, E.,
(2011). Dataset-driven Research for Improving Recommender Systems for Learning. Learning
Analytics & Knowledge: February 27-March 1,17 2011, Banff, Alberta, Canada

32. dataTEL::Body of knowledge
Outcomes:
Tanimoto similarity +
item-based CF was
the most accurate.
Outcomes:
Implicit ratings like download
rates, bookmarks can
successfully used in TEL.
Verbert, K., Drachsler, H., Manouselis, N., Wolpers, M., Vuorikari, R., Beham, G., Duval, E.,
(2011). Dataset-driven Research for Improving Recommender Systems for Learning. Learning
Analytics & Knowledge: February 27-March 1,17 2011, Banff, Alberta, Canada

33. Goals of the lecture
1.Motivation or dataTEL
2.The dataTEL project
3.Potentials of dataTEL
4.Open Issues of dataTEL
18

34. Potentials of Open Data
Example by Tim Berners-Lee: The year open data went worldwide, TED talk FEB 2010
19

35. Potentials of Open Data
Example by Tim Berners-Lee: The year open data went worldwide, TED talk FEB 2010
19

36. Data = New Science Paradigm
• Thousand years ago science was
empirical (Describing natural
phenomena)
20

37. Data = New Science Paradigm
• Thousand years ago science was
empirical (Describing natural
phenomena)
• Last few hundred years science:
theoretical branch (Using
models, generalizations)
20

38. Data = New Science Paradigm
• Thousand years ago science was
empirical (Describing natural
phenomena)
• Last few hundred years science:
theoretical branch (Using
models, generalizations)
• Last few decades: computational
branch (Simulating complex phenomena)
20

39. Data = New Science Paradigm
• Thousand years ago science was
empirical (Describing natural
phenomena)
• Last few hundred years science:
theoretical branch (Using
models, generalizations)
• Last few decades: computational
branch (Simulating complex phenomena)
• Nowadays: data science
(Unify theory, experiment, and simulation,
data captured by instruments and processed
by software, linked data)
20

40. Promises of Open Data for TEL
21

41. Promises of Open Data for TEL
Unexploited potentials for TEL:
• The evaluation of learning theories and learning
technology from the data side
• More transparent, mutually comparable, trusted and
repeatable experiments that lead to evidence-driven
knowledge
• Development of new educational data tools / products
that combine different data sources in data mashups
• Gain new insights / new knowledge by combining so far
unconnected resources / tools
21

42. Data Products
22

43. Data Products
22

44. Data Products
22

45. Data Products
W. Reinhardt, C. Mletzko, H. Drachsler, and P. Sloep. AWESOME: A widget-based
dashboard for awareness-support in Research Networks. In Proceedings of the 2nd PLE
Conference, Southampton, UK, 2011.
22

46. Data Products
Educational Data Products
• Drop-out Analyzer
• Group Formation Recommender
• Question-Answering Tool
• Awareness Tools
W. Reinhardt, C. Mletzko, H. Drachsler, and P. Sloep. AWESOME: A widget-based
dashboard for awareness-support in Research Networks. In Proceedings of the 2nd PLE
Conference, Southampton, UK, 2011.
22

47. Goals of the lecture
1.Motivation or dataTEL
2.The dataTEL project
3.Potentials of dataTEL
4.Open issues of dataTEL
23

48. dataTEL::Open issues
1.Privacy
2.Prepare datasets
3.Share datasets
4.Body of knowledge
24

49. Privacy
25

50. Privacy
25

51. Privacy
OVERSHARING
25

52. Privacy
OVERSHARING
Were the founders of PleaseRobMe.com actually
allowed to take the data from the web and present it
in that way?
25

53. Privacy
OVERSHARING
Were the founders of PleaseRobMe.com actually
allowed to take the data from the web and present it
in that way?
Are we allowed to use data from social services and
reuse it for research purposes?
25

54. Privacy
26

55. Privacy
1.Privacy as confidentiality
The right to be let alone (Warren and Brandeis, 1890)
26

56. Privacy
1.Privacy as confidentiality
The right to be let alone (Warren and Brandeis, 1890)
2.Privacy as control
The right of the individual to decide what information
about herself should be communicated to others and
under which circumstances.
26

57. Privacy
1.Privacy as confidentiality
The right to be let alone (Warren and Brandeis, 1890)
2.Privacy as control
The right of the individual to decide what information
about herself should be communicated to others and
under which circumstances.
3.Privacy as practice
The right to intervene in the flows of existing data and
the re-negotiation of boundaries with respect to
collected data.
26

58. Privacy solutions
27

59. Privacy solutions
1.Privacy as confidentiality
Information services that minimizing, secure or
anonymize the collected information
27

60. Privacy solutions
1.Privacy as confidentiality
Information services that minimizing, secure or
anonymize the collected information
2.Privacy as control
Identity Management Systems (IDMS),
with access control rules
27

61. Privacy solutions
1.Privacy as confidentiality
Information services that minimizing, secure or
anonymize the collected information
2.Privacy as control
Identity Management Systems (IDMS),
with access control rules
3.Privacy as practice
Timestamp on data, data degradation technologies
27

62. Prepare datasets
Justin Marshall, Coded Ornament by
rootoftwo
http://www.flickr.com/photos/rootoftwo/
267285816
28

63. Prepare datasets
1. Create a dataset that
realistically reflects the
variables of the learning
setting.
Justin Marshall, Coded Ornament by
rootoftwo
http://www.flickr.com/photos/rootoftwo/
267285816
28

64. Prepare datasets
1. Create a dataset that
realistically reflects the
variables of the learning
setting.
2. Use a sufficiently large
set of user profiles
Justin Marshall, Coded Ornament by
rootoftwo
http://www.flickr.com/photos/rootoftwo/
267285816
28

65. Prepare datasets
1. Create a dataset that
realistically reflects the
variables of the learning
setting.
2. Use a sufficiently large
set of user profiles
3. Create datasets that
are comparable to others
Justin Marshall, Coded Ornament by
rootoftwo
http://www.flickr.com/photos/rootoftwo/
267285816
28

66. Prepare datasets
For informal data sets:
1. Collect data
2. Process data
3. Document data
4. Share data
For formal data sets
from LMS:
1. Data storing scripts
2. Anonymisation scripts
3. Document data
4. Share data
29

67. Prepare datasets
30

68. Share/cite datasets
31

69. Sharing policies
32

70. Sharing policies
32

71. Sharing policies
32

72. Sharing policies
32

73. Sharing policy guidelines
A brief guide on data licenses developed by SURF and the Centre for
Intellectual Property Law (CIER), 2009 available at
www.surffoundation.nl
33

74. Body of knowledge
Datasets
Formal Informal
Data A Data B Data C
Algorithms: Algorithms: Algorithms:
Algoritmen A Algoritmen D Algoritmen B
Algoritmen B Algoritmen E Algoritmen D
Algoritmen C
Models: Models: Models:
Learner Model A Learner Model C Learner Model A
Learner Model B Learner Model E Learner Model C
Measured attributes: Measured attributes: Measured attributes:
Attribute A Attribute A Attribute A
Attribute B Attribute B Attribute B
Attribute C Attribute C Attribute C
34

75. Body of knowledge
35

76. Body of knowledge
35

77. Body of knowledge
35

78. Body of knowledge
35

79. Body of knowledge
35

80. dataTEL::SIG
http://www.teleurope.eu/pg/groups/9405/datatel/
Objectives:
• Representing dataTEL researchers to promote the release
of open datasets from educational providers
• Fostering the standardizations of datasets to enable
exchange and interoperability
• Contributing to policies on ethical guidelines (privacy and
legal protection rights)
• Fostering a shared understanding of evaluation methods in
TEL RecSys and Learning Analytics technologies.
36

81. Many thanks for your interests
37
picture by Tom Raftery http://www.flickr.com/photos/traftery/4773457853/sizes/l

82. Many thanks for your interests
Free
the data
37
picture by Tom Raftery http://www.flickr.com/photos/traftery/4773457853/sizes/l

83. Many thanks for your interests
This silde is available at:
http://www.slideshare.com/Drachsler
Email: hendrik.drachsler@ou.nl
Skype: celstec-hendrik.drachsler
Blogging at: http://www.drachsler.de
Twittering at: http://twitter.com/HDrachsler
38