Learning analytics is a 'hot topic' in education with many institutions seeking to make better use of the data available via various systems. One of the key challenges in this process is to understand the business questions that people working in various roles in institutions would like to be able to answer. However, it is also important that these questions are appropriately structured and specific in order to gather the relevant data. This session builds on the workshop run at last year's Blackboard Learning and Teaching conference where participants explored business questions and use cases for learning analytics from a range of perspectives.
Delivered at Innovate and Educate: Teaching and Learning Conference by Blackboard. 24 -27 August 2015 in Adelaide, Australia.
Explore beautiful and ugly buildings. Mathematics helps us create beautiful d...
Moving Forward on Learning Analytics - A/Professor Deborah West, Charles Darwin University and Ms Nicole Wall, Blackboard ANZTLC15
1. Moving Forward on Learning
Analytics
A/Professor Deborah West,
Charles Darwin University
Ms Nicole Wall, Blackboard
2. Today’s session
• Brief background
• Review the information gathered on the business questions from last
year
• Present an analysis of which of these questions can be answered via
specific Analytics for Learn reports
• Review a set of reports to check their validity in relation to the use
case
• Explore the development of ‘good’ questions which can be used to
further refine reporting.
2
3. ‘Learning Analytics is about
collecting traces that learners
leave behind and using those to
improve learning’ (Duval, 2012)
3
4. Our Focus
• collecting traces that students and staff leave behind and using
those to improve the student experience
4
5. Different types of data and affordances
• Functionality of LMS
• Analytics packages for the LMS
• Data warehouses
5
6. Benefits of Learning Analytics
Able to provide data to:
• Create a more structured and effective intervention to improve student
outcomes
• Understand workload planning
• Value add over time through student tracking
• Improve support, target training workshops
• Identify successful strategies and evaluate teacher effectiveness
• Strategy and planning
6
8. Curriculum/Course Design Questions
• How active are students in course content overall and at an
individual content level?
• What types of content are included in courses? How does this relate
to what is most actively used and the results students achieve?
• Can I couple student’s ratings of content/activities with their use of
that content / activities with their results?
• Is there a correlation between different course ‘styles’ (e.g.
instructivist vs. constructivist) and student achievement within a
given course or faculty?
8
9. Student Behaviour Questions
• When and how are students engaged with the LMS and associated
tools?
– what times/lengths of day
– spread of access throughout academic calendar, particularly key
milestone dates?
– what types of devices and where from?
• When did a student last access email? When did a student last access
Wi-Fi?
• Is there correlation between messaging and test attempts?
9
10. Retention Questions
• What kind of pre-enrolment interactions lead to “better”
outcomes for our students?
• Analytics for identifying, monitoring, managing and responding
to students at risk.
• What are the indicators?
• Who are the most at risk?
• What interventions are more likely to be effective?
• What trends can we see in terms of at risk students
10
11. Institutional level planning/resourcing
• Gauging impact of new initiatives/introduction of new
technologies
• Where are students connecting with “the university” in their
learning journey?
• Is the time and/or money/effect that has been put into
something worthwhile? Are there particular things which are
over or underrepresented?
11
12. Academic Behaviours
• Which academics and which “schools” need support to become
better/fuller users of Bb Learn?
• What’s the frequency or % of academic attention per student within
a subject?
• How engaged are academics with the full Bb Learn toolset?
• How much time instructors are spending and where the peaks of
activity are?
• Usability and Ease of Access by lecturers.
12
13. Miscellaneous Questions
• Is this data: reliable, meaningful, and accurate?
• If we have the “answer” are we morally, ethically and legally
required to act?
• How much inference can be drawn from the level of
engagement a student has with an online environment?
13
14. Getting Answers from Analytics for Learn
14
Questions Themese #Qs Report
Available
Report Can be
Built
No Direct Match
Curriculum/Course design &
improvement
19 47% 21% 31%
Student behaviour 7 42% 57%
Improve Retention 10 100%
Resourcing/planning at institutional
level
8 38% 13% 50%
Academic actions/behaviour 6 50% 33% 17%
Questions from Different Perspectives 20 60% 40%
15. Writing good questions for your reports
• Focus on Questions That Are Aligned With Your Business
Strategy
• Keep your questions specific and related to opportunities for
applying the insights
• Be definitive - One of the most valuable questions asked is how
can you engage your students better? But what is your
definition of ‘engagement’.
15
16. Writing good questions for your reports
16
Grade &
Outcome
Results
Course
Attributes
Student
Attributes
Final
Grades
Student
Information
System
Reports
Instructo
r
AttributesUser
Activit
y Data
Course
Item
Data
17. Writing good questions for your reports
• What are you trying to measure?
– Interactions, Accesses, Minutes, Submissions
• How are you trying to measure it?
– Average, Percentage, Count
• To what?
– Blackboard, Courses, Items
17
18. Writing good questions for your reports
• How do you want to filter or group the information
– Course Attributes
– Instructor Attributes
– Student Attributes
– Performance
– Time Frame
18
19. Writing good questions for your reports
19
How many logins occurred in Semester 1 and 2 by Faculty
Which Faculty had the highest logins in the last 2 semester
How is engagement with Blackboard changing
✔
✗
✔
20. Activity
• Refine the question
– Be specific; incorporate the measures and dimensions that are available
– Consider the purpose of the question – what you want to know and for
what purpose
– Might break down into more than one
• On the butchers paper draw what the report would look like
– E.g. graph, table, pie chart
20
21. But…
“Everything that can be counted does
not necessarily count; everything
that counts cannot necessarily be
counted.”
Albert Einstein
21
Hinweis der Redaktion
Course Design – hard to measure most of the questions e.g best way to structure the content –how do we measure this?
Student Behavior – many questions outside the scope of A4L e.g. email, device, IP
Improve Retention - Analytics can help to track indicators/those at risk/if interventions change behaviour but it does not define them
Resource Planning – many questions were about how to use the data not what you want to answer
Why and how to do it
Specific - It has to be clear what the KPI exactly measures. There has to be one widely-accepted definition of the KPI to make sure the different users interpret it the same way and, as a result, come to the same and right conclusions which they can act on.
Measurable - The KPI has to be measurable to define a standard, budget or norm, to make it possible to measure the actual value and to make the actual value comparable to the budgeted value.
Achievable - Every KPI has to be measurable to define a standard value for it. It is really important for the acceptance of KPI’s and Peformance Management in general within the organization that this norm is achievable. Nothing is more discouraging than striving for a goal that you will never obtain.
Relevant - The KPI must give more insight in the performance of the organization in obtaining its strategy. If a KPI is not measuring a part of the strategy, acting on it doesn’t affect the organizations’ performance. Therefore an irrelevant KPI is useless.
Time phased - It is important to express the value of the KPI in time. Every KPI only has a meaning if one knows the time dimension in which it is realized. The realization and standardization of the KPI therefore has to be time phased.
Why and how to do it
Measures are the first step
Why and how to do it
Dimesnions help us to slice and dice te information
Why and how to do it
Dimesnions help us to slice and dice