Data Analytics process in Learning and Academic Analytics projects. Day 3: Data processing

1. Data Analytics process in
Learning and Academic
Analytics projects
Day 3: Data processing
Alex Rayón Jerez
alex.rayon@deusto.es
DeustoTech Learning – Deusto Institute of Technology – University of Deusto
Avda. Universidades 24, 48007 Bilbao, Spain
www.deusto.es

2. Table of contents
● Data dimensions
● Applications
● Data processing in an ETL refined data
● Knowledge discovery

3. Table of contents
● Data dimensions
● Applications
● Data processing in an ETL refined data
● Knowledge discovery

4. Data dimensions
Summary
[Verbert2011]

5. Data dimensions
1) Computing
● Software
○ Example
■ Q1. Among the tools, which is more representative of
the final grade?
■ Q5. Which is the impact of the social networks in the
group composition?
■ Q6. Which tools are more prone to foster
collaboration?
■ Q7. The use of some collaboration tools has effect on
the final grade?
● Hardware
● Network

6. Data dimensions
2) Location
● Quantitative
● Qualitative

7. Data dimensions
3) Time
● Timestamp
● Time interval

8. Data dimensions
4) Activity
● Events
● Tasks
● Goals
● Subject
○ Example
■ Q2. Which are the differences in terms of grades
between this subject and other subjects where we
already know the final grade?

9. Data dimensions
5) Physical condition
● Noise level
● Lighting
● ...

10. Data dimensions
6) Resource
● Physical resource
● Virtual resource

11. Data dimensions
7) User
● Basic info
○ Example
■ Q3. Is there any gender difference in the use of the
tools?
● Knowledge
● Interest
● Goals
○ Short-term
○ Long-term
● Learning styles
● Affects
● Background

12. Data dimensions
8) Relations
● Social relations
○ Example
■ Q4. Are there groups of people that repeatedly
collaborate in different tools?
■ Q4. Do these groups repeat over time?
● Functional relations
● Compositional relations
● Proximity
● Orientation
● Communication

13. Table of contents
● Data dimensions
● Applications
● Data processing in an ETL refined data
● Knowledge discovery

14. Applications
Why do learners use analytics?
[Ferguson2014]
● Monitor their own activities and interactions
● Monitor the learning process
● Compare their activity with that of others
● Increase awareness, reflect and self reflect
● Improve discussion participation
● Improve learning behaviour
● Improve performance
● Become better learners
● Learn!

15. Applications
Why do teachers use analytics?
[Ferguson2014]
● Monitor the learning process
● Explore student data
● Identify problems
● Discover patterns
● Find early indicators for success
● Find early indicators for poor marks or drop-
out
● Assess usefulness of learning materials

16. Applications
Why do teachers use analytics? (Ii)
● Increase awareness, reflect and self reflect
● Increase understanding of learning
environments
● Intervene, advise and assist
● Improve teaching, resources and the
environment

17. Table of contents
● Data dimensions
● Applications
● Data processing in an ETL refined data
● Knowledge discovery

18. Data processing
Transform menu

19. Data processing
Scripting menu

20. Data processing
Joins menu

21. Data processing
Statistics menu

22. Data processing
WEKA plugin

23. Data processing
WEKA plugin (II)

24. Data processing
WEKA plugin (III)

25. Data processing
WEKA plugin (IV)

26. Data processing
WEKA plugin (V)

27. Table of contents
● Data dimensions
● Applications
● Data processing in an ETL refined data
● Knowledge discovery

28. Knowledge discovery
Introduction
[BakerSiemens2014]
This review draws on past reviews (cf. Baker & Yacef, 2009; Romero & Ventura, 2010;
Ferguson, 2012; Siemens & Baker, 2012)

29. Knowledge discovery
Introduction (II)
Source: Data Mining with WEKA MOOC (http://www.cs.waikato.ac.nz/ml/weka/mooc/dataminingwithweka/)

30. Knowledge discovery
Classification
1. Prediction methods
2. Structure discovery
3. Relationship mining

31. Knowledge discovery
1) Prediction methods
● The goal is to develop a model which can infer
a single aspect of the data
○ The predicted variable
○ Similar to dependent variables in traditional statistical
analysis
● … from some combination of other aspects
of the data
○ Predictor variables
○ Similar to independent variables in traditional
statistical analysis

32. Knowledge discovery
1) Prediction methods (II)
● Prediction models are commonly used:
○ Predict future events (Dekker2009; Feng2009;
MingMing2012)
○ Predict variables that are not feasible to directly
collect in real-time
■ Example: collecting data on affect or engagement in
real-time often requires expensive observations or
disruptive self-report measures
■ Whereas a prediction model based on student log
data can be completely non-intrusive
(Sabourin2011)

33. Knowledge discovery
1) Prediction methods (III)
Source: http://etec.ctlt.ubc.ca/510wiki/Learning_Analytics

34. Knowledge discovery
1) Prediction methods (IV)
● Three types of prediction models are common
in EDM/LA:
○ Classifiers
○ Regressors
○ Latent knowledge estimation

35. Knowledge discovery
1) Prediction methods (V)
Source: Data Mining with WEKA MOOC (http://www.cs.waikato.ac.nz/ml/weka/mooc/dataminingwithweka/)

36. Knowledge discovery
1) Prediction methods (VI)
● Classifiers
○ The predicted variable can be either a binary (e.g. 0 or
1) or a categorical variable
○ Some popular classification methods in educational
domains include:
■ Decision trees
■ Random forest
■ Decision rules
■ Step regression
■ Logistic regression

37. Knowledge discovery
1) Prediction methods (VII)
Source: Data Mining with WEKA MOOC (http://www.cs.waikato.ac.nz/ml/weka/mooc/dataminingwithweka/)

38. Knowledge discovery
1) Prediction methods (VIII)
● Regressors
○ The predicted variable is a continuous variable
■ For example: if the Grade can be explained by the
number of pending subjects and the call number
○ The most popular regressor in EDM is linear
regression
■ Note that linear regression is not used the same
way in EDM/LA as in traditional statistics, despite
the identical name

39. Knowledge discovery
1) Prediction methods (IX)
Source: Data Mining with WEKA MOOC (http://www.cs.waikato.ac.nz/ml/weka/mooc/dataminingwithweka/)

40. Knowledge discovery
1) Prediction methods (X)
Source: Data Mining with WEKA MOOC (http://www.cs.waikato.ac.nz/ml/weka/mooc/dataminingwithweka/)

41. Knowledge discovery
1) Prediction methods (XI)
● Latent Knowledge Estimation
○ Actually is a special type of classifier
○ A student’s knowledge of specific skills and concepts is
assessed by their patterns of correctness on those
skills
○ A wide range of algorithms exist for latent knowledge
estimation, being the two most popular:
■ Bayesian Knowledge Tracing (Corbett & Anderson,
1995)
■ Performance Factors Analysis (Pavlik2009)

42. Knowledge discovery
1) Prediction methods (XII)
● Classifiers in WEKA are models for predicting
nominal or numeric quantities
● Implemented learning schemes include:
○ Decision trees and lists, instance-based classifiers,
support vector machines, multi-layer perceptrons,
logistic regression, Bayes’ nets, etc.
● “Meta”-classifiers include:
○ Bagging, boosting, stacking, error-correcting output
codes, locally weighted learning, etc.

43. Knowledge discovery
1) Prediction methods (XIII)

44. Knowledge discovery
2) Structure discovery
● Attempt to find structure in the data without
an a priori idea of what should be found
● It is, actually, a very different goal than in
prediction
○ In prediction, there is a specific variable that the
EDM/LA researcher attempts to model;
○ By contrast, there is not a specific variable of
interest in structure discovery
○ Instead, the researcher attempts to determine what
structure emerges naturally from the data

45. Knowledge discovery
2) Structure discovery (II)
● Include:
○ Clustering
○ Factor analysis
○ Social Network Analysis
○ Domain Structure Discovery

46. Knowledge discovery
2) Structure discovery (III)
● Clustering
○ The goal is to find data points that naturally group
together, splitting the full data set into a set of clusters
○ Clustering is particularly useful in cases where the
most common categories within the data set are not
known in advance
○ If a set of clusters is well-selected, each data point in a
cluster will generally be more similar to the other data
points in that cluster than data points in other clusters

47. Knowledge discovery
2) Structure discovery (IV)
● Clustering
○ Clusters have been used to group students (Beal2006)
and student actions (Amershi2009)
■ Amershi & Conati (2009) found characteristic
patterns in how students use exploratory learning
environments, and used this information to
identify more and less effective student strategies

48. Knowledge discovery
2) Structure discovery (IV)
● Factor analysis
○ A closely related method
○ Here, the goal is to find variables that naturally group
together, splitting the set of variables (as opposed to
the data points) into a set of latent (not directly
observable) factors
○ Factor analysis is frequently used in psychometrics for
validating or determining scales

49. Knowledge discovery
2) Structure discovery (V)
● Factor analysis
○ In EDM/LA, factor analysis is used for dimensionality
reduction (e.g., reducing the number of variables) for a
wide variety of applications
○ For instance, [Baker2009] used factor analysis to
determine which design choices are made in common
by the designers of intelligent tutoring systems
■ For instance, tutor designers tend to use principle
based hints rather than concrete hints in tutor
problems that have brief problem scenarios

50. Knowledge discovery
2) Structure discovery (VI)
● Social Network Analysis
○ Models are developed of the relationships and
interactions between individual actors, as well as the
patterns that emerge from those relationships and
interactions
○ Examples
■ Understanding the differences between effective
and ineffective project groups [Kay2006]
■ How students’ communication behaviors change
over time [Haythornthwaite2001]
■ How students’ positions in a social network relate
to their perception of being part of a learning

51. Knowledge discovery
2) Structure discovery (VII)
● Domain structure discovery
○ Consists of finding the structure of knowledge in an
educational domain (e.g., how specific content maps
to specific knowledge components or skills, across
students)
○ This could consist of mapping problems in educational
software to specific knowledge components, in order
to group the problems effectively for latent knowledge
estimation and problem selection [Koedinger2006], or
could consist of mapping test items to skills
[Tatsuoka1995]

52. Knowledge discovery
2) Structure discovery (VIII)
● WEKA contains “clusterers” for finding groups
of similar instances in a dataset
● Implemented schemes are:
○ k-Means, EM, Cobweb, X-means, FarthestFirst
● Clusters can be visualized and compared to
“true” clusters (if given)
● Evaluation based on loglikelihood if clustering
scheme produces a probability distribution

53. Knowledge discovery
3) Relationship mining
● Discover relationships between variables in a
data set with a large number of variables
● It has historically been the most common
category of EDM research [Baker2009]
● It may take the form of attempting to find out
which variables are most strongly associated
with a single variable of particular interest
● Or may take the form of attempting to
discover which relationships between any two
variables are strongest

54. Knowledge discovery
3) Relationship mining (II)
● There are four types of relationship mining
○ Association rule mining
○ Correlation mining
○ Sequential pattern mining
○ Causal data mining

55. Knowledge discovery
3) Relationship mining (III)
● Association rule mining
○ The goal is to find if-then rules of the form that if some
set of variable values is found, another variable will
generally have a specific value
○ For instance, [BenNaim2009] used association rule
mining to find patterns of successful student
performance in an engineering simulation, to make
better suggestions to students having difficulty about
how they can improve their performance

56. Knowledge discovery
3) Relationship mining (IV)
● Correlation mining
○ The goal is to find positive or negative linear
correlations between variables (using post-hoc
corrections or dimensionality reduction methods
when appropriate to avoid finding spurious
relationships)
○ An example can be found in [Baker2009], where
correlations were computed between a range of
features of the design of intelligent tutoring system
lessons and students’ prevalence of gaming the system

57. Knowledge discovery
3) Relationship mining (V)
● Sequential pattern mining
○ The goal is to find temporal associations between
events
○ One successful use of this approach was work by
[Perera2009], to determine what path of student
collaboration behaviors leads to a more successful
eventual group project

58. Knowledge discovery
3) Relationship mining (VI)
● Causal data mining
○ The goal is to find whether one event (or observed
construct) was the cause of another event (or
observed construct)
○ For example to predict which factors will lead a
student to do poorly in a class [Fancsali2012]

59. Knowledge discovery
3) Relationship mining (VII)
● WEKA contains an implementation of the
Apriori algorithm for learning association
rules
○ Works only with discrete data
● Can identify statistical dependencies between
groups of attributes:
○ milk, butter bread, eggs (with confidence 0.9 and
support 2000)
● Apriori can compute all rules that have a given
minimum support and exceed a given
confidence

60. Knowledge discovery
3) Relationship mining (VIII)

61. Knowledge discovery
4) Attribute selection
● Panel that can be used to investigate which
(subsets of) attributes are the most
predictive ones
● Attribute selection methods contain two parts:
○ A search method: best-first, forward selection,
random, exhaustive, genetic algorithm, ranking
○ An evaluation method: correlation-based, wrapper,
information gain, chi-squared, etc.
● Very flexible: WEKA allows (almost) arbitrary
combinations of these two

62. Knowledge discovery
4) Attribute selection (II)

63. Knowledge discovery
4) Attribute selection (III)

64. References
[Amershi2009] Amershi, S., Conati, C. (2009). Combining Unsupervised and Supervised Machine Learning to Build User Models for Exploratory
Learning Environments. Journal of Educational Data Mining, 1(1), 71-81.
[BakerSiemens2014] Baker, R., and George Siemens. "Educational data mining and learning analytics." Cambridge Handbook of the Learning
Sciences: (2014).
[BakerYacef2009] Baker, R.S.J.d., Yacef, K. (2009). The State of Educational Data Mining in 2009: A Review and Future Visions. Journal of
Educational Data Mining, 1 (1), 3-17
[Beal2006] Beal, C.R., Qu, L., & Lee, H. (2006). Classifying learner engagement through integration of multiple data sources. Paper presented at the
21st National Conference on Artificial Intelligence (AAAI-2006), Boston, MA.
[CorbettAnderson1995] Corbett, A.T., Anderson, J.R. (1995). Knowledge Tracing: Modeling the Acquisition of Procedural Knowledge. User
Modeling and User-Adapted Interaction, 4, 253-278.
[Dawson2008] Dawson, S. (2008). A study of the relationship between student social networks and sense of community. Educational Technology &
Society, 11(3), 224-238.
[Dekker2009] Dekker, G., Pechenizkiy, M., and Vleeshouwers, J. (2009). Predicting students drop out: A case study. Proceedings of the 2nd
International Conference on Educational Data Mining, EDM'09, 41-50
[Fancsali2012] Fancsali, S. (2012) Variable Construction and Causal Discovery for Cognitive Tutor Log Data: Initial Results. Proceedings of the 5th
International Conference on Educational Data Mining, 238-239.
[Feng2009] Feng, M., Heffernan, N., & Koedinger, K. (2009). Addressing the Assessment Challenge in an Intelligent Tutoring System that Tutors as it
Assesses. User Modeling and User-Adapted Interaction, 19, 243-266
[Ferguson2012] Ferguson, R. (2012). The State Of Learning Analytics in 2012: A Review and Future Challenges. Technical Report KMI-12-01,
Knowledge Media Institute, The Open University, UK. http://kmi.open.ac.uk/publications/techreport/kmi-12-01
[Ferguson2014] Learning analytics FAQs [Online]. URL: http://www.slideshare.net/R3beccaF/learning-analytics-fa-qs
[Haythornthwaite2001] Haythornthwaite, C. (2001). Exploring Multiplexity: Social Network Structures in a ComputerSupported Distance Learning
Class. The Information Society: An International Journal, 17 (3), 211-226.
[Kay2006] Kay, J., Maisonneuve, N., Yacef, K., Reimann, P. (2006) The Big Five and Visualisations of Team Work Activity. Proceedings of the
International Conference on Intelligent Tutoring Systems, 197 – 206.

65. References (II)
[Koedinger2006] Koedinger, K. R., & Corbett, A. T. (2006). Cognitive Tutors: Technology bringing learning science to the classroom. In K. Sawyer
(Ed.) The Cambridge Handbook of the Learning Sciences (pp. 61-78). New York: Cambridge University Press.
[MingMing2012] Ming, N.C., Ming, V.L. (2012). Predicting Student Outcomes from Unstructured Data. Proceedings of the 2nd International
Workshop on Personalization Approaches in Learning Environments, 11-16.
[Pavlik2009] Pavlik, P.I., Cen, H., Koedinger, K.R. (2009). Performance Factors Analysis -- A New Alternative to Knowledge Tracing. Proceedings of
AIED2009.
[Perera2009] Perera, D., Kay, J., Koprinska, I., Yacef, K., and Zaiane, O.R. (2009). Clustering and Sequential Pattern Mining of Online Collaborative
Learning Data. IEEE Transactions on Knowledge and Data Engineering, 21(6), 759-772
[RomeroVentura2010]Romero, C., & Ventura, S. (2010). Educational data mining: A review of the state-ofthe-art. IEEE Transaction on Systems,
Man and Cybernetics, part C: Applications and Reviews, 40(6), 610–618
[Sabourin2011] Sabourin, J., Rowe, J., Mott, B., Lester, J. (2011). When Off-Task in On-Task: The Affective Role of Off-Task Behavior in Narrative-
Centered Learning Environments. Proceedings of the 15th International Conference on Artificial Intelligence in Education, 534-536.
[SiemensBaker2012] Siemens, G., Baker, R.S.J.d. (2012). Learning Analytics and Educational Data Mining: Towards Communication and
Collaboration. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge.
[Tatsuoka1995] Tatsuoka, K.K. (1995). Architecture of knowledge structures and cognitive diagnosis: A statistical pattern recognition and
classification approach. In P. D. Nichols, S. F. Chipman, & R. L. Brennan (Eds.), Cognitively diagnostic assessment, 327–359. Hillsdale NJ: Erlbaum
[Verbert2011] Dataset-driven research to improve TEL recommender systems [Online]. URL: http://www.slideshare.net/kverbert/datasetdriven-
research-to-improve-tel-recommender-systems

66. Data Analytics process in
Learning and Academic
Analytics projects
Day 3: Data processing
Alex Rayón Jerez
alex.rayon@deusto.es
DeustoTech Learning – Deusto Institute of Technology – University of Deusto
Avda. Universidades 24, 48007 Bilbao, Spain
www.deusto.es