Python Notes for mca i year students osmania university.docx
Data Analytics.01. Data selection and capture
1. Data Analytics process in
Learning and Academic
Analytics projects
Day 1: Data selection and capture
Alex Rayón Jerez
alex.rayon@deusto.es
DeustoTech Learning – Deusto Institute of Technology – University of Deusto
Avda. Universidades 24, 48007 Bilbao, Spain
www.deusto.es
2. Objectives
How to tackle an AA/LA project
1. Objectives: what do I want to improve?
2. Data: automated processes for data discovery
and later processing
3. Integration, not substitution
4. Technology
5. KPIs: define and test
3. Table of contents
● ETL approach
● Data analytics cycle
● Architecture principles
● Requirements
● Components
● Data process
○ Questions
○ Data model
○ Data sources
○ Use cases
4. Table of contents
● ETL approach
● Data analytics cycle
● Architecture principles
● Requirements
● Components
● Data process
5. ETL approach
Definition and characteristics
● An ETL tool is a tool that
○ Extracts data from various data sources (usually
legacy data)
○ Transforms data
■ from → being optimized for transaction
■ to → being optimized for reporting and analysis
■ synchronizes the data coming from different
databases
■ data cleanses to remove errors
○ Loads data into a data warehouse
6. ETL approach
Why do I need it?
● ETL tools save time and money when
developing a data warehouse by removing
the need for hand-coding
● It is very difficult for database administrators
to connect between different brands of
databases without using an external tool
● In the event that databases are altered or new
databases need to be integrated, a lot of hand-
coded work needs to be completely redone
8. ETL approach
Kettle (II)
● It uses an innovative meta-driven approach
● It has a very easy-to-use GUI
● Strong community of 13,500 registered
users
● It uses a stand-alone Java engine that
process the tasks for moving data between
many different databases and files
12. ETL approach
Kettle (VI)
● Datawarehouse and datamart loads
● Data integration
● Data cleansing
● Data migration
● Data export
● etc.
13. ETL approach
Transformations
● String and Date Manipulation
● Data Validation / Business Rules
● Lookup / Join
● Calculation, Statistics
● Cryptography
● Decisions, Flow control
● Scripting
● etc.
14. ETL approach
What is good for?
● Mirroring data from master to slave
● Syncing two data sources
● Processing data retrieved from multiple
sources and pushed to multiple
destinations
● Loading data to RDBMS
● Datamart / Datawarehouse
○ Dimension lookup/update step
● Graphical manipulation of data
15. Table of contents
● ETL approach
● Data analytics cycle
● Architecture principles
● Requirements
● Components
● Data process
16. Data Analytics cycle
Challenges
● Data is everywhere
● Data is inconsistent
○ Records are different in each system
● Performance issues
○ Running queries to summarize data for
stipulated long period takes operating
system for task
○ Brings the OS on max load
● Data is never all in Data Warehouse
○ Excel sheet, acquisition, new application
17. Data Analytics cycle
Challenges (II)
● Data is incomplete
● Certain types of usage data are not logged
● Data are not aggregated following a
didactical perspective
● Users are afraid that they could draw
unsound inferences from some of the data
[Mazza2012]
18. Data Analytics cycle
Academic Analytics Model
1) Capture
2) Report5) Refine
4) Act 3) Predict
Academic Analytics
[CampbellOblinger2007]
19. Data Analytics cycle
Learning Analytics Model
1) Select
2) Capture
3) Aggregate
4) Process
5) Visualize
On the design of collective applications
[DronAnderson2009]
20. Data Analytics cycle
Learning Analytics Model (II)
On the design of collective applications
[DronAnderson2009]
1) Select
2) Capture
3) Aggregate
4) Process
5) Visualize
Day 1
Day 2
Day 3
Day 4
21. Data Analytics cycle
Learning Analytics Model (III)
● As [Clow2012] states, it is
necessary to close the
feedback loop through
appropriate interventions
unmistakable
● It also draws on the wider
educational literature,
seeking to place learning
analytics on an established
theoretical base, and
develops a number of
insights for learning
analytics practice
24. Table of contents
● ETL approach
● Data analytics cycle
● Architecture principles
● Requirements
● Components
● Data process
25. Architecture principles
A model for adoption, use and improvement of analytics
A framework of characteristics for Analytics
Adam Cooper, 2012 [Cooper2012]
26. Architecture principles
Development of common language for data exchange
The IEEE defines interoperability to be:
“The ability of two or more systems or
components to exchange information and
to use the information that has been
exchanged”
28. Architecture principles
Development of common language for data exchange (III)
● The most difficult challenges with achieving
interoperability are typically found in
establishing common meanings to the data
● Sometimes this is a matter of technical
precision
○ But culture – regional, sector-specific, and
institutional – and habitual practices also affect
meaning
29. Architecture principles
Development of common language for data exchange (IV)
● Potential benefits
○ Efficiency and timeliness
■ No need for a persona to intervene to re-enter, re-
format or transform data
○ Independence
■ Resilience
○ Adaptability
■ Faster, cheaper and less disruptive to change
○ Innovation and market growth
■ Interoperability combined with modularity makes
it easier to build IT systems that are better
matched to local culture without needing to create
30. Architecture principles
Development of common language for data exchange (V)
● Potential benefits
○ Durability of data
■ Structures and formats change over time
■ The changes are rarely properly documented
○ Aggregation
■ Data joining might be supported by a common set
of definitions around course structure, combined
with a unified identification scheme
○ Sharing
■ Specially when there are multiple parties involved
34. Table of contents
● ETL approach
● Data analytics cycle
● Architecture principles
● Requirements
● Components
● Data process
35. Requirements
● Usability: prepare an understandable user interface
(UI), appropriate methods for data visualization, and
guide the user through the analytics process.
● Usefulness: provide relevant, meaningful indicators
that help teachers to gain insight in the learning
behavior of their students and support them in
reflecting on their teaching.
● Interoperability: ensure compatibility for any kind
of VLE by allowing for integration of different data
sources.
[Dyckhoff2010]
36. Requirements (II)
● Extensibility: allow for incremental extension of
analytics functionality after the system has been
deployed without rewriting code.
● Reusability: target for a building-block approach to
make sure that re-using simpler ones can implement
more complex functions.
● Real-time operation: make sure that the toolkit can
return answers within microseconds to allow for an
exploratory user experience
● Data Privacy: preserve confidential user information
and protect the identities of the users at all times
[Dyckhoff2010]
37. Table of contents
● ETL approach
● Data analytics cycle
● Architecture principles
● Requirements
● Components
● Data process
38. Components
● Process
○ A systematic process of educational data analysis
● Model
○ The definition of a suitable model to represent the
knowledge domain
● Tool/platform
○ The design and implementation of a monitoring
and presentation tool based on the Process and
Model
[Mazza2012]
39. Table of contents
● ETL approach
● Data analytics cycle
● Architecture principles
● Requirements
● Data process
40. Data process
Introduction
“Measurement, collection, analysis and
reporting of data about learners and
their contexts, for purposes of
understanding and optimising learning
and the environments in which it
occurs”
First international conference on Learning Analytics and
Knowledge, Alberta, 2011 [LAK2011]
41. Data process
Introduction (II)
However, the challenge is to determine
which data are of interest
We are now in an era where gaining
access to data is not the problem;
the challenge lies in determining
which data are significant and why
42. Data process
Introduction (III)
“The basic question
is not what can we
measure? The basic
question is what
does a good
education look like?
Big questions”
43. Data process
Introduction (IV)
“More data does not mean more knowledge”
[Jenkins2013]
Searching for the evidence in a mass of data
requires knowing what kind of evidence is
needed
Knowledge of the domain and understanding
and interpretation of the patterns we see
47. Data process
Introduction (VIII)
First of all, education is a highly collaborative space and it represents a social good. Keeping a valuable secret
that might help students succeed is antithetical to the nature of education. Second, education is a complex
ecosystem of people, processes, policies, content, etc. I would have strong doubts about anyone who claimed to
have a formula that worked for a wide variety of institutions.
Mike Sharkey, 2014
51. Data process
Questions (IV)
1) Adaptive testing, tracking and reporting
● Progress summary, daily activity report, class
goals report, progress report, student activity
report, student focus report, etc [Khan2012]
● By using various analytics tools, students can
review their learning progress and teachers
are also supported in how to personalise
learning for students in need for more help in
specific areas
52. Data process
Questions (V)
2) Analytics tools for early alert, intervention
and collaboration
● Integrating their data collected from a variety
of information management systems
○ Allowing educators to assess the risk, initiate early
interventions and support collaborative learning
53. Data process
Questions (VI)
2) Analytics tools for early alert, intervention
and collaboration
● For example, the Signals project at Purdue
University utilizes the data collected from
student information systems, learning
management systems, and the grade book for
a specific course to track students’
performances and identify at-risk students in
real time
54. Data process
Questions (VII)
2) Analytics tools for early alert, intervention
and collaboration
● The LOCO-Analyst provides teachers with
charts, graphs, and other data representations
that help them see how their students are
performing and how students interact with
one another in web-based learning
environments to help the teacher determine
how to engage their students online
55. Data process
Questions (VIII)
2) Analytics tools for early alert, intervention
and collaboration
● Social Networks Adapting Pedagogical
Practice (SNAPP), a network visualization tool
developed by researchers at the University of
Wollongong, can analyse students’
interactions in a forum and display it in a
visualised diagram which help teachers to
identify the key connections and disconnected
students and support collaborative learning in
a web-based learning environment
56. Data process
Questions (IX)
3) Analytics projects for institutional
efficiency and effectiveness
● There are a number of institutional analytics
initiatives which enable institutions to
improve the effectiveness of operations,
including admission management and drop-
out prevention, resource management,
financial planning, etc
○ Student Experience Traffic Lighting (SETL)
○ The Enhancing Student Centred Administration
Placement Experience (ESCAPES)
60. Data process
Questions (XIII)
● The Harvard and MIT data ignores student
goals or any information giving a clue on
whether students desired to complete the
course, get a good grade, get a certificate, or
just sample some material
● Without this information, the actual
aggregate behavior is missing context
○ We don’t know if a certain student intended to just
audit a course, sample it, or attempt to complete it.
○ We don’t know if students started the course intended
to complete but became frustrated
61. Data process
Questions (XIV)
● The value of learner behavior patterns, which
can only be learned by viewing data patterns
over time
● If you want to “share best practices to improve
teaching and learning”, then you need data
organized around the learner
○ With transactions captured over time – not just in
aggregate
○ What we have now is an honest start, but a very
limited data set
63. Data process
Data model (II)
The data model, or the concept map,
describes the concepts and their
relationships used by the organization
in its daily work, expressed in its own
language
It enables the whole organization to
participate in the maintenance of it
64. Data process
Data model (III)
Source: http://www.economist.com/news/finance-and-economics/21578041-containers-have-been-more-important-globalisation-freer-trade-
humble
Source: http://www.economist.com/blogs/economist-explains/2013/05/economist-explains-14
65. Data process
Data model (IV)
The best approach that we have
found for this task is constituted by
the theory of eLearning functions
Reinmann [Reinmann2006]
68. Data process
Data model (VII)
Example
This model answers the monitoring questions:
● Which way of eLearning enables to reach the
given objectives?
● By which means (functions, tools) does the
LMS enable these ways of learning?
● How is the use of these means traced in the log
files (activity log codes)?
[Mazza2012]
70. Data process
Data sources
Today we have so much data
that come in an unstructured
or semi-structured form that
may nonetheless be of value in
understanding more about our
learners
71. Data process
Data sources (II)
“Learning is a complex social activity”
[Siemens2012]
Lots of data
Lots of tools
Humans to make sense
72. Data process
Data sources (III)
Traditional data sources:
● Student data: demographics,
qualification aim, modules taken,
results, etc.
● Student feedback data: end of
module survey and others
● Student activity data: delivery data,
completion, pass rates, etc.
73. Data process
Data sources (IV)
● The world of technology has changed
[Eaton2012]
○ 80% of the world’s information is unstructured
○ Unstructured data are growing at 15 times the rate
of structured information
○ Raw computational power is growing at such an
enormous rate that we almost have a supercomputer
in our hands
○ Access to information is available to all
74. Data process
Data sources (V)
Source: http://www.bigdata-startups.com/BigData-startup/understanding-sources-big-data-infographic/
75. Data process
Data sources (VI)
● RDBMS (SQL Server, DB2, Oracle, MySQL,
PostgreSQL, Sybase IQ, etc.)
● NoSQL Data: HBase, Cassandra, MongoDB
● OLAP (Mondrian, Palo, XML/A)
● Web (REST, SOAP, XML, JSON)
● Files (CSV, Fixed, Excel, etc.)
● ERP (SAP, Salesforce, OpenERP)
● Hadoop Data: HDFS, Hive
● Web Data: Twitter, Facebook, Log Files, Web Logs
● Others: LDAP/Active Directory, Google Analytics,
etc.
79. Data process
Use cases (II)
2) Moodle: MySQL database
mdl_forum
- id
- course
- name
mdl_user
- id
- username
- firstname
- lastname
mdl_forum_discussions
- id
- name
- userid
- timemodified
- usermodified
mdl_forum_posts
- id
- userid
- discussion
- message
- modified
- created
80. Data process
Use cases (III)
3) MediaWiki: MySQL database
user
- user_real_name
- user_editcount recentchanges
- rc_old_len
- rc_new_len
revision
- rev_timestamp page
- page_counter
- page_len
rev_user = user_id
rev_page = page_id
user_id = rc_user
92. References
[CampbellOblinger2007] Campbell, John P., Peter B. DeBlois, and Diana G. Oblinger. "Academic analytics: A new tool for a new era." Educause
Review 42.4 (2007): 40.
[Clow2012] Clow, Doug. "The learning analytics cycle: closing the loop effectively." Proceedings of the 2nd International Conference on Learning
Analytics and Knowledge. ACM, 2012.
[Cooper2012] Cooper, Adam. "What is analytics? Definition and essential characteristics." CETIS Analytics Series 1.5 (2012): 1-10.
[DronAnderson2009] Dron, J., & Anderson, T. (2009). On the design of collective applications. In Proceedings of the 2009 International Conference
on Computational Science and Engineering, 4, 368–374.
[Dyckhoff2010] Dyckhoff, Anna Lea, et al. "Design and Implementation of a Learning Analytics Toolkit for Teachers." Educational Technology &
Society 15.3 (2012): 58-76.
[Eaton2012] Chris Eaton, Dirk Deroos, Tom Deutsch, George Lapis & Paul Zikopoulos, “Understanding Big Data: Analytics for Enterprise Class
Hadoop and Streaming Data”, p.XV. McGraw-Hill, 2012.
[GayPryke2002] Cultural Economy: Cultural Analysis and Commercial Life (Culture, Representation and Identity series) Paul du Gay (Editor),
Michael Pryke. 2002.
[HR2012] NMC Horizon Report 2012 http://www.nmc.org/publications/horizon-report-2012-higher-ed-edition
[Jenkins2013] BBC Radio 4, Start the Week, Big Data and Analytics, first broadcast 11 February 2013 http://www.bbc.co.
uk/programmes/b01qhqfv
[Khan2012] http://www.emergingedtech.com/2012/04/exploring-the-khan-academys-use-of-learning-data-and-learning-analytics/
[LACE2013] Learning Analytics Community Exchange http://www.laceproject.eu/
[LAK2011] 1st International Conference on Learning Analytics and Knowledge, 27 February - 1 March 2011, Banff, Alberta, Canada https://tekri.
athabascau.ca/analytics/
[Mazza2006] Mazza, Riccardo, et al. "MOCLog–Monitoring Online Courses with log data." Proceedings of the 1st Moodle Research Conference. 2012.
[Reinmann2006] Reinmann, G. (2006). Understanding e-learning: an opportunity for Europe? European Journal of Vocational Training, 38, 27-42.
[SiemensBaker2012] Siemens & Baker (2012). Learning Analytics and Educational Data Mining: Towards Communication and Collaboration.
Learning Analytics and Knowledge 2012. Available in .pdf format at http://users.wpi.edu/~rsbaker/LAKs%20reformatting%20v2.pdf
93. Data Analytics process in
Learning and Academic
Analytics projects
Day 1: Data selection and capture
Alex Rayón Jerez
alex.rayon@deusto.es
DeustoTech Learning – Deusto Institute of Technology – University of Deusto
Avda. Universidades 24, 48007 Bilbao, Spain
www.deusto.es