Presented at ALT-C 2016 In our increasingly digital world our students leave an ever-growing electronic footprint behind them as they pass through the physical and virtual campus. This data is still a greatly underused asset (Higher Education Commission, 2016) although a number of UK higher education institutions have already implemented descriptive, inferential and/or predictive Learning Analytics (LAs) solutions using a wide variety of approaches, see for example Sclater (2014). This paper discusses recent research at an English post-92 university aimed at investigating the benefits and challenges of using LAs. Prompted by a perception that some voices had yet to be given a loud enough voice in relation to a systematic use of big data in the higher education sector. It was particularly concerned with gaining a better understanding of the hopes, fears and needs of those on whom it would be most likely to impact. This presentation will focus on our findings from of a series of focus groups and interviews with students, university governors and academic professional and support staff that took place during the 2015/16 academic year. Questions were framed around understanding views about the purpose of LAs, concerns about the type of data liable to be used, perceptions of how likely being more informed would result in changes in behaviour and outcomes, and finally how should data be presented. The results indicated that each group had different areas of interest when it came to the type of data of interest. These spanned the range from what may be regarded as strict LAs to Academic Analytics (for an explanation of the differences see Long and Siemens (2014)). A common theme however, was that most felt that being better informed would lead to better decision making. However, having knowledge about one’s own performance, particularly in relation to peers, was unwelcome in some quarters. When examining student concerns about data, overall there was a low degree of anxiety and a high degree of trust that the institution in general and tutors in particular would behave responsibly. Concerns about legal and ethical problems were most likely to be voiced by academic, professional and support staff. Transparency, and finding creative approaches to promoting it, was identified as vital by most groups. During this session we will share our findings in more detail and reflect on our understanding of variations in perceptions between and within different stakeholder groups. We will demonstrate and share a checklist of institutional risks and responsibilities that was developed as an aid to identifying, understanding and managing each of these areas.

1. @PlymUniASTI /PlymUniASTIasti@plymouth.ac.ukwww.plymouth.ac.uk/asti
Learning Analytics: What Do Stakeholders Really Think?
Prof Neil Witt, Dr Anne McDermott, Prof Pauline Kneale & Prof David Coslett

2. Academic Support, Technology & Innovation
 ASTI and Teaching and Learning Support
investigating the potential of using Learning
Analytics as a means of enhancing the student
experience
 Grant from the HEA’s Strategic Excellence Initiative
for Vice-Chancellors
Background
Teaching and Learning Support

3. Academic Support, Technology & Innovation
 The electronic footprint our students leave behind when they
interact with our digital systems
e.g. digital learning environment, electronic library, ePortfolio
 This and other data sources used to track student engagement
and to identify those who may be in danger of failing
 Web pages and apps used to present various data visualisations
for personal tutors, students and others
Teaching and Learning Support
Definition of Learning Analytics

4. Academic Support, Technology & Innovation
Teaching and Learning Support
Definition of Learning Analytics
Institution Faculty School Programme Module Individual
Academic
Analytics
Learning
Analytics
Data
Granularity of Data

5. Academic Support, Technology & Innovation
Teaching and Learning Support
A student perspective?

6. Academic Support, Technology & Innovation
Teaching and Learning Support
Understanding our Stakeholders’ Perspectives

7. Academic Support, Technology & Innovation
 Mostly information already collected in a range of ways,
from a range of systems for a range of purposes
 Array of challenges           
 Systematic identification of ‘at risk’ students may place an
unsustainable obligation to act on the University
Challenges:
 Ethical
 Legal
 Data
 Technical
 Policy
 Process
Teaching and Learning Support
Key issues, challenges and concerns

8. Academic Support, Technology & Innovation
 Transparency vital to maintain trust
 High degree of confidence that staff
would deal with their data in a
‘professional way’
 Varied in degree of comfort with easy
access to data about their own
performance - closely monitored
students seemed least worried e.g. health
areas
 Most, but not all, keen to compare their
profile with anonymised cohort average or
an ‘ideal’ student, so consider ‘opt out/in’
 Could motivate some but discourage others
 University’s response to a ‘red flag’ should
not be an automatic process but the start of
a conversation
Teaching and Learning Support
Stakeholder Perspectives:
Students

9. Academic Support, Technology & Innovation
 More concerns than students over legal /
ethical data use
 Support and training to understand
responsibilities
 Need for openness and transparency
 Focus should be on benefitting student
not institution
 Questioned effect on retention students leave
for many reasons
 Policy changes needed, e.g. attendance
monitoring
 Much of this data is already collected in
disparate ways
 Need to ensure compliance with legislation
for current, retrospective and future use of
data
Teaching and Learning Support
Stakeholder Perspectives:
Academic, Technical & Support Staff

10. Academic Support, Technology & Innovation
 Need shared vision of what is meant
by Learning Analytics
 Culture of respect for information and
anonymity required
 Concerns about the scope and quality
of current data
 Analytics data should be triangulated
with other information
 Opposing views about students having own and
cohort data
‘why would they want to know?’
‘what are they afraid of?’
 Some concerns it could be demotivational, a
distraction or encourage a strategic approach to
study
 Could give insight into characteristics of a
successful programme, trajectory of a successful
student, value added over the course of a
programme (Learning Gain)
Teaching and Learning Support
Stakeholder Perspectives:
Senior Leaders

11. Academic Support, Technology & Innovation
 Potential to increase retention and enhance performance but would need to
show a return on investment
 Varied responses to having personal access
 With Analytics, data must become everyone’s responsibility
 Academic Analytics could aid institutional decision-making
 Enable Plymouth to offer something distinctive to its students
Teaching and Learning Support
Stakeholder Perspectives:
Governors

12. Academic Support, Technology & Innovation
Teaching and Learning Support
Recommendations

13. Academic Support, Technology & Innovation
 LA owned at a very high level
 Plan for success e.g. fewer
withdrawals
 Define goal(s) and specify initial
measures
 Audit policies to identify
amendments and gaps
 Implement single version of
truth for data & policies
 Set and resource Institution-
wide standards for responding
 Build-in to future procurements
 Consider offering ‘opt-out’
Teaching and Learning Support
Recommendations: Policy

14. Academic Support, Technology & Innovation
 Consent agreements and
statements in line with planned use
 Choice of Learning Analytics
solution
 Implement institution-wide
standards for responding
 Governance requires a
multidisciplinary team including
students
 Digital literacy & training for staff
 Be open and transparent,
particularly with students
 Be aware there will be false
negatives/ false positives
 Staff development to make
responsibilities clear and
support policy changes
Teaching and Learning Support
Recommendations: Process

15. Academic Support, Technology & Innovation
 Bring silos together (e.g. data
warehouse)
 Single version of truth needed
for chosen data
 Specify data currently easily
accessible
 Establish ownership,
stewardship and users of data
 Agreements with 3rd party
provider to reflect new use
 Unique identifier work required
 Work out synergies with other
existing projects (e.g. S3, Mobile
With Plymouth app)
Teaching and Learning Support
Recommendations: Technology

16. Academic Support, Technology & Innovation
 A response to a ‘red flag’ should not be
an automatic process but the start of a
conversation
 Implement single version of truth for
data and policies
 Policy changes needed, e.g. attendance
monitoring
 Support and training to understand
responsibilities and support policy
changes
 Culture of respect for information
and anonymity required
 Establish ownership, stewardship and
users of data
 Policies and data need to be owned
centrally (i.e. Academic Registry)
 With Analytics, data must become
everyone’s responsibility
Teaching and Learning Support
Institutional Checklist

17. Academic Support, Technology & Innovation
 Senior Sponsorship is essential
 Single Version of the truth for all data and policies
 Have an “owner” for data and revised/updated/new policies
 Use available solutions (i.e. Jisc toolkit, Mobile With Plymouth, S3)
 Use analytics to support personal tutoring and institutional decision making
 Learning Analytics is about Culture Change, not technology
Teaching and Learning Support
Making it so

18. Academic Support, Technology & Innovation
Thank you
Questions?