Learning Analytics 101
•Download as PPTX, PDF•
5 likes•4,406 views
Learning Analytics is an emerging topic of interest throughout all levels of education focusing on how to harness the power of data mining, interpretation, and modeling. However, there are several similar terms (academic analytics, predictive analytics, business intelligence, etc.) that can confuse educators and administrators alike. In this session, we will unpack this new area of interest and discuss how institutions can begin to leverage available products and open source communities to utilize analytics to improve understandings of teaching and learning and to tailor education more effectively. We will briefly present an overview of the learning analytics field, drawing from popular examples such as the Signals project at Purdue U. and the Check My Activity tool at U. Maryland, Baltimore County. We will also review the structure of Sakai CLE and OAE user-level metrics and briefly discuss projects to design and implement tools to utilize these metrics in meaningful ways.
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (11)
Similar to Learning Analytics 101
Similar to Learning Analytics 101 (20)
More from Steven Lonn
More from Steven Lonn (8)
Recently uploaded
Recently uploaded (20)
Learning Analytics 101
- 1. Steve Lonn, University of Michigan Josh Baron, Marist College June 10-15, 2012 Growing Community; Growing Possibilities
- 2. 1. What is Learning Analytics (LA)? 2. Current LA work in Higher Education 3. Data available in Sakai CLE & OAE 4. Big Questions to Ponder 5. Q&A Slides Available: slideshare.net/stevelonn/ 2012 Jasig Sakai Conference 2
- 3. “...datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze.” Manyika et al. (2011) 2012 Jasig Sakai Conference 3
- 4. 2012 Jasig Sakai Conference 4
- 5. Analytics: An overarching concept that is defined as data-driven decision making van Barneveld, Arnold, & Campbell, 2012 adapted from Ravishanker 2012 Jasig Sakai Conference 5
- 6. 2012 Jasig Sakai Conference 6
- 7. Business / Academic Analytics: A process for providing higher education institutions with the data necessary to support operational and financial decision making van Barneveld, Arnold, & Campbell, 2012 adapted from Goldstein and Katz 2012 Jasig Sakai Conference 7
- 8. evidenceframework.org/big-data/ Educational Data Mining Learning Analytics Bienkowski, Feng, & Means, 2012 ◦ SRI International 2012 Jasig Sakai Conference 8
- 9. Generally emphasizes reduction into small, easily analyzable components ◦ Can be then adapted to student by software ◦ Siemens and Baker, 2012 Predicting future learning behavior Domain models for content / sequences Software-provided pedagogical supports Computational models that incorporate student, domain, and pedagogy 2012 Jasig Sakai Conference 9
- 10. Example: Cognitive Tutors Pittsburgh Advanced Cognitive Tutor Center Carnegie Mellon University http://ctat.pact.cs.cmu.edu 2012 Jasig Sakai Conference 10
- 11. Educational Data Mining: A process for analyzing data collected during teaching and learning to test learning theories and inform educational practice Bienkowski, Feng, & Means, 2012 2012 Jasig Sakai Conference 11
- 12. Understand entire systems and support human decision making Applies known methods & models ◦ answer questions about learning and organizational learning systems Tailored responses ◦ adapted instructional content, specific interventions, providing specific feedback 2012 Jasig Sakai Conference 12
- 13. Learning Analytics: The use of analytic techniques to help target instructional, curricular, and support resources to support the achievement of specific learning goals through applications that directly influence educational practice van Barneveld, Arnold, & Campbell, 2012 adapted from Bach 2012 Jasig Sakai Conference 13
- 14. Predictive Analytics ◦ uncover relationships and patterns ◦ can be used to predict behavior and events Visual Data Analytics ◦ discovering and understanding patterns in large datasets via visual interpretation 2012 Jasig Sakai Conference 14
- 15. Term Definition Level of Focus An overarching concept that is defined as data- Analytics All levels driven decision making A process for providing higher education Academic institutions with the data necessary to support Institution Analytics operational and financial decision making A process for analyzing data collected during Department / Educational teaching and learning to test learning theories Instructor / Data Mining Learner and inform educational practice The use of analytic techniques to help target instructional, curricular, and support resources Department / Learning Instructor / to support the achievement of specific learning Analytics Learner goals through applications that directly influence educational practice 2012 Jasig Sakai Conference 15
- 16. Who‟s been working in this space in Higher Education? 2012 Jasig Sakai Conference 16
- 17. Purdue University‟s Course Signals ◦ College-wide learning analytics approach University of Michigan‟s E2Coach ◦ Course-specific learning analytics approach UMBC‟s “Check My Activity” Tool ◦ Student-centered learning analytics approach 2012 Jasig Sakai Conference 17
- 18. Built predictive model using data from… ◦ LMS – Events (login, content, discuss.) & gradebook ◦ SIS – Aptitude (SAT/ACT, GPA) & demographic data Leverage model to create Early-alert system ◦ Identify students at risk to not complete the course ◦ Deploy intervention to increase chances of success Systems automates intervention process ◦ Students get “traffic light” alert in LMS ◦ Messages are posted to student that suggest corrective action (practice tests) 2012 Jasig Sakai Conference 18
- 19. Impact on course grades and retention ◦ Students in courses using Course Signals… scored up to 26% more A or B grades up to 12% fewer C's; up to 17% fewer D's and F„s Ellucian product that integrates w/Blackboard Open Academic Analytics Initiative (OAAI) ◦ Creating a similar Sakai-based OS solution Arnold & Pistilli, 2012 - LAK 2012 Jasig Sakai Conference 19
- 20. Focused specifically on introductory Physics Uses data from… ◦ Pre-course survey: academic info, learner‟s goals, psycho-social factors ◦ Performance: Exams, Web HW, Sakai Michigan Tailoring System (MTS) ◦ OS tool designed for highly customized messaging ◦ Used in health sciences for behavior change ◦ Messaging based on input from many sources “…to say to each what we would say if we could sit down with them for a personal chat.” 2012 Jasig Sakai Conference 20
- 21. 2012 Jasig Sakai Conference 21
- 22. UMBC found that students earning D/F‟s use Bb 39% less then higher grade achievers ◦ Not suggesting cause and effect ◦ Goal is to model higher achiever behavior Provides data directly student ◦ Compare LMS use to class averages ◦ Can also compare averages usage data to grade outcomes Feedback has been positive 2012 Jasig Sakai Conference 22
- 23. Student Success Plan – Sinclair CC ◦ Holistic case-management system ◦ Connects faculty, advisors, counselors, & students ◦ Jasig Incubation Project STAR Academic Journey – U of Hawaii ◦ Online advising and degree attainment system SNAPP – UBC/Wollongong ◦ Visualize networks of interaction resulting from discussion forum posts and replies 2012 Jasig Sakai Conference 23
- 24. Papers and Articles on Purdue‟s Course Signals http://www.itap.purdue.edu/learning/research/ Michigan‟s Expert Electronic Coaching http://sitemaker.umich.edu/ecoach/home UMBC‟s Check My Activity Tool http://www.educause.edu/EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazineVolum/Vi deoDemoofUMBCsCheckMyActivit/219113 Student Success Plan http://www.educause.edu/EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazineVol um/TheStudentSuccessPlanCaseManag/242785 STAR Academy Journey http://net.educause.edu/ir/library/pdf/pub7203cs7.pdf SNAPP http://research.uow.edu.au/learningnetworks/seeing/snapp 2012 Jasig Sakai Conference 24
- 25. What can we know in CLE and OAE products? 2012 Jasig Sakai Conference 25
- 26. User-level data stored as “events” sakai_event sakai_session EVENT_ID SESSION_ID EVENT_DATE SESSION_USER EVENT SESSION_IP REF SESSION_USER_AGENT SESSION_ID SESSION_START EVENT_CODE SESSION_END CONTEXT SESSION_SERVER SESSION_ACTIVE SESSION_HOSTNAME List of events available on Confluence ◦ Search for “event table description” 2012 Jasig Sakai Conference 26
- 27. Site-level data stored in separate tables sakai_site sakai_realm CUSTOM_PAGE_ORDERED REALM_KEY SITE_ID REALM_ID TITLE PROVIDER_ID TYPE MAINTAIN_ROLE SHORT_DESC CREATEDBY DESCRIPTION MODIFIEDBY ICON_URL CREATEDON INFO_URL MODIFIEDON SKIN PUBLISHED JOINABLE PUBVIEW JOIN_ROLE CREATEDBY MODIFIEDBY CREATEDON realm_id like '/site/' || site_id MODIFIEDON IS_SPECIAL IS_USER 2012 Jasig Sakai Conference 27
- 28. 16,000 14,000 12,000 10,000 8,000 6,000 4,000 2,000 0 F04 F05 F06 F07 F08 F09 F10 F11 W05 W06 W07 W08 W09 W10 W11 W12 Project Sites Course Sites Max Users Thanks to John Leasia 2012 Jasig Sakai Conference 28
- 29. 3% 2% 1% 1%1% Presence 3% Web Content Resources Attachments 43% Test Center 18% Assignments Syllabus Forums Gradebook Drop Box 25% Evaluations 2012 Jasig Sakai Conference 29
- 30. Social Work Architecture Engineering Business LS&A Education Public Health Art & Design Law Nursing Music Medicine Dentistry Pharmacy 0% 25% 50% 75% 100% 2012 Jasig Sakai Conference 30
- 31. Clinical Assoc Prof Clinical Lecturer Clinical Professor Clinical Asst Prof Asst Professor Professor Assoc Professor Asst Professor Adjunct Clin Asst Professor Adjunct Clinical Lecturer Adjunct Clin Assoc Prof 0% 25% 50% 75% 100% 2012 Jasig Sakai Conference 31
- 32. BIT Sum of Revisions ENGR NURS IOE SI ENGLISH RCHUMS AAAS EECS RCLANG Count of Course Sites SI ENGLISH BIT EECS PSYCH COMP MODGREEK NURS NRE 2012 Jasig Sakai Conference 32
- 33. Summary information about site visits, tool activity, and resource activity 2012 Jasig Sakai Conference 33
- 34. 2012 Jasig Sakai Conference 34
- 35. User-level data available via “activity feeds” ◦ follows a “push and publication” model rather than a “store and query” model (CLE is store & query) ◦ Activity is both highly specific: individual interactions between users, content, contexts… ◦ …and more general: user interaction everywhere rather than only within a single course context. What new questions will we ask? ◦ Interesting activity can happen with external capabilities: CLE tools, LTI tools, widgets. How will we ensure this data is captured? Many thanks to Nate Angell for OAE slides 2012 Jasig Sakai Conference 35
- 36. FYI: Designs are still in draft form. 2012 Jasig Sakai Conference 36
- 37. FYI: Designs are still in draft form. 37 2012 Jasig Sakai Conference
- 38. FYI: Designs are still in draft form. 38 2012 Jasig Sakai Conference
- 39. Activity (OAE) & Grades (CLE): Week 1 Developed by the Kaleidoscope Project in collaboration with rSmart. 2012 Jasig Sakai Conference 39
- 40. Activity (OAE) & Grades (CLE): Week 7 Developed by the Kaleidoscope Project in collaboration with rSmart. 2012 Jasig Sakai Conference 40
- 41. Activity (OAE) & Grades (CLE): Animation Developed by the Kaleidoscope Project in collaboration with rSmart. 2012 Jasig Sakai Conference 41
- 42. Tools / services to support analytics initiatives ◦ Ways to connect different silos of data ◦ Methods to connect back to CLE / OAE LTI? Web services? Others? OAE improvements over CLE approach to user data ◦ What data is most relevant for analytics? ◦ What displays and/or data are most useful to help learners? 2012 Jasig Sakai Conference 42
- 43. Josh Baron, Marist College 2012 Jasig Sakai Conference 43
- 44. Data Mining vs. Learning Science Approaches ◦ Do we build predictive models from large data sets or from our understanding of learning sciences? ◦ Is both the right answer? How does that work? Challenges of Scaling LA Across Higher Ed ◦ Does each institution have to build its own model? How “portable” are predictive models? ◦ Do we need an open standard for LA? Could LIS and LTI play a role? How can LA be used to assist ALL students? ◦ Michigan‟s E2Coach system is a good example 2012 Jasig Sakai Conference 44
- 45. “The obligation of knowing” – John Campbell ◦ If we have the data and tools to improve student success, are we obligated to use them? Consider This > If a student has a 13% chance of passing a course, should they be dropped? 3%? Who owns the data, the student? Institution? ◦ Should students be allowed to “opt out”? Consider This > Is it fair to the other students if by opting out the predictive model‟s power drops? What do we reveal to students? Instructors? Consider This > If we tell a student in week three they have a 9% chance of passing, what will they do? Will instructors begin to “profile” students? 2012 Jasig Sakai Conference 45
- 46. Connect with Learning Analytics communities 2012 Jasig Sakai Conference 46
- 47. http://www.solaresearch.org/ Learning Analytics & Knowledge Conferences (LAK) STORM – initiative to help fund research projects FLARE – regional practitioner conference ◦ Purdue University, Oct 1-3, 2012 2012 Jasig Sakai Conference 47
- 48. Symposium on Learning Analytics at Michigan http://sitemaker.umich.edu/slam/ 15 speakers (12 UM, 3 external) Videos & slides available from all speakers 2012 Jasig Sakai Conference 48
- 49. Analytics in Higher Education: Establishing a Common Language ◦ Van Barneveld, Arnold, Campbell, 2012 ◦ http://www.educause.edu/Resources/AnalyticsinHigherEducationEsta/245405 Analytics to Literacies: Emergent Learning Analytics to evaluate new literacies ◦ Dawson, 2011- http://blogs.ubc.ca/newliteracies/files/2011/12/Dawson.pdf Learning Analytics: Definitions, Process Potential ◦ Elias, 2011 ◦ http://learninganalytics.net/LearningAnalyticsDefinitionsProcessesPotential.pdf The State of Learning Analytics in 2012: A Review and Future Challenges ◦ Ferguson, 2012 - http://kmi.open.ac.uk/publications/pdf/kmi-12-01.pdf Academic analytics: A new tool for a new era. ◦ Campbell, Deblois, & Oblinger (2007). Educause Review, 42(4), 40-57. ◦ http://net.educause.edu/ir/library/pdf/ERM0742.pdf Mining LMS data to develop an "early warning system" for educators: A proof of concept. ◦ Macfadyen & Dawson (2010) - Computers & Education, 54(2), 588-599. Classroom walls that talk: Using online course activity data of successful students to raise self- awareness of underperforming peers. ◦ Fritz, 2011 - Internet and Higher Education, 14(2), 89-97. 2012 Jasig Sakai Conference 49
- 50. Wednesday, 13 June ◦ Learning Analytics: A Panel Debate on the Merits, Methodologies, and Related Issues (1:15pm) ◦ Learning Analytics at Michigan: Designing Displays for Advisors, Instructors, and Students (2:30pm) ◦ BOF for Learning Analytics: Current and Planned Projects and Tools (3:45pm) Thursday, 14 June ◦ Creating an Open Ecosystem for Learner Analytics (10:15am) Open Academic Analytics Initiative (OAAI) https://confluence.sakaiproject.org/x/8aWCB 2012 Jasig Sakai Conference 50
- 51. Steve Lonn ◦ slonn@umich.edu @stevelonn Josh Baron ◦ Josh.baron@marist.edu @joshbaron 2012 Jasig Sakai Conference 51
Editor's Notes
- Headlines from THIS MONTH that mention Big Data\\But Big Data is not just Google, Facebook, Twitter, Amazon, and Netflix…
- Student Information SystemsLearning Management SystemsBroad-Based SurveysPublisher journals & eTextbooksVideo contentLecture captureEportfoliosAnd more…
- Student Information SystemsLearning Management SystemsBroad-Based SurveysPublisher journals & eTextbooksVideo contentLecture captureEportfoliosAnd more…
- HousingAdmissionsFinancial AidHuman ResourcesDepartments & Units’ own metrics
- HousingAdmissionsFinancial AidHuman ResourcesDepartments & Units’ own metrics
- Generally, educational data mining is looking for new patterns in data and developing new algorithms and/or new models, while learning analytics is applying known predictive models in instructional systems.
- Common Questions for EDM:What sequence of topics is most effective for a specific student? What student actions are associated with more learning (e.g., higher course grades)? What student actions indicate satisfaction, engagement, learning progress, etc.? What features of an online learning environment lead to better learning? What will predict student success?
- Math tutor – gives hints where stuck, data about progress can inform next question type & difficulty
- When are students ready to move onto the next topic? When are students falling behind in a course? When is a student at risk for not completing a course? What grade is a student likely to get without intervention? What is the best next course for a given student? Should a student be referred to a counselor for help?
- When are students ready to move onto the next topic? When are students falling behind in a course? When is a student at risk for not completing a course? What grade is a student likely to get without intervention? What is the best next course for a given student? Should a student be referred to a counselor for help?
- Josh’s Section
- Steve’s Section
- REF changes depending on event typeAt Michigan and other institutions of similar size, both of these tables are archived and saved into non-production copies, so only the last 24-36 hours of data is typically located on the production servers. This also allows analytics work to occur without impacting production.
- Most tools also have their own tables with relevant data such as Resources, Gradebook, Forums, etc. Due to the open-source nature of these tools, the structure of the data and how to query relevant information is often difficult and not uniform across tools
- Here is a graph of different site types and maximum number of concurrent users per term that is calculated from data in the sakai_site tableCan be useful for institutional decisions about servers, space, etc.
- Query using information from the HR database matched against the sakai_session tableWinter 2010 data
- Winter 2010 data
- Generate reports based on: site visits tool events resource activityNarrow search by:specific tools, events or resourcespre-defined or custom time periodsall/specific users, roles, groups or non-matching usersExcel, CSV and PDF file export
- Generate reports based on: site visits tool events resource activityNarrow search by:specific tools, events or resourcespre-defined or custom time periodsall/specific users, roles, groups or non-matching usersExcel, CSV and PDF file export
- Activity feeds will be the next feature that is added to OAE, and is currently already in development. They will show user activity from different perspectives (e.g. all activity relevant to me, all activity related to a course, etc.) and include things like someone uploading a file, someone commenting on a file, a person you follow updating his profile, someone participating in a discussion, etc.Whilst CLE has often collected quite technical user-level information in the sakai-event table, OAE is taking an approach where it tries to separate technical events/activity from user level events/activity. Some of these user-level events will be captured and stored automatically, and perhaps shipped off to a data warehouse for analytics. But an important part of OAE analytics will also be the ability for end-users to determine what they want to capture for a given context (e.g. How long was the student here", "How many minutes of the video did he watch" and "After how many questions did this student stop filling out this quiz"). This would be done using a JavaScript API and a UI wrapper for that API, and that data can then also be fed into the analytics process.
- If we determine a student is “at risk”, are we obligated to intervene?
- Also MOOCs, new Journal, etc.
- Great way to start conversation at your own institution and connect faculty & researchers who are working with student data, but may not think they are doing learning analytics
- Also session this morning about work at University of HullAnd sessions about Jasig projects like Student Success Plan (Tuesday morning)