Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Support assessment and feedback practice with technology: a view of the UK landscape
1. Supporting assessment and feedback
practice with technology: a view of the UK
landscape
Marianne Sheppard and Dr Gill Ferrell, Jisc infoNet
EUNIS 2013
2. UK Context
Area of highest student
dissatisfaction - National Student
Survey scores
Drivers to enhance the student
experience vs. increasing resource
constraints
Workload burden – academic and
administrative
18 June, 2013 EUNIS Conference 2013, Riga slide 2
3. Addressing the challenges
Jisc Assessment and Feedback programme
(2011-2014)
20 projects and 25 institutions involved across
the UK
3 strands focused on institutional
change, evaluation of technologies and
software development
Supporting large-scale changes to assessment
and feedback practice through technology
18 June, 2013 EUNIS Conference 2013, Riga slide 3
www.jisc.ac.uk/assessmentandfeedback
4. Strategy and policy
Infrastructure
Assessment and feedback practice
Engagement
18 June, 2013 EUNIS Conference 2013, Riga slide 4
Assessment and feedback challenges
8. University of Westminster: Reflecting on feedback
18 June, 2013 EUNIS Conference 2013, Riga slide 8
‘It has helped I
think because
since then my
marks have shot
up.”’
12. Manchester Metropolitan University: Assessment Lifecycle
18 June, 2013 EUNIS Conference 2013, Riga slide 12
Manchester Metropolitan University Assessment Lifecycle (TRAFFIC project)
13. MMU: e-Submission
18 June, 2013 EUNIS Conference 2013, Riga slide 13
Manchester Metropolitan University Assessment Lifecycle (TRAFFIC project)
14. University of Huddersfield: e-Submission
Benefits
Increased control and agency
Reduced anxiety
Improved privacy and security
Increased efficiency and convenience
Feedback which is clearer and easier to
engage with, understand and access for
later use
18 June, 2013 EUNIS Conference 2013, Riga slide 14
University of Huddersfield (EBEAM project)
15. Open approaches and standards
18 June, 2013 EUNIS Conference 2013, Riga slide 15
University of Nottingham (Rogō OSS project)
Kingston University (Uniqurate project)
16. 18 June, 2013 EUNIS Conference 2013, Riga slide 16
From principles to practiceManaging
change
Good afternoon and welcome to this presentation where I will be talking to you about some of the UK sector issues and challenges around assessment and feedback and how these can be supported by technology.
Assessment and feedback is at the heart of the student learning experience and forms a significant part of the academic and administrative workload. However it continues to be the biggest area of student dissatisfaction according to the UK’s annual national student survey. As a result higher education providers are looking to enhance the way learners are supported and make the management of assessment more efficient in a climate of increasing resource constraints. This paper will look at the work of the Jisc Assessment and Feedback programme, a 3 year programme supporting institution-wide change and explore different approaches that projects are taking to using technology to enhance assessment and feedback processes and practices.
Jisc, a UK body supporting the use of digital technologies in UK higher education, further education and skills, is funding a this work involving over 20 UK higher education institutions.There are 3 strands focusing on institutional change, evaluation of technologies in use and development technology transfer. Overall there is a strong emphasis the educational rationale for enhancing learning and teaching through technology and delivering efficiencies and quality improvements.
A number of the institutions in the programme conducted a detailed review of assessment and feedback practice which revealed a rich picture of the current state of play across the sector. Whilst there are excellent examples of effective support for learning and the use of technology to support these, there is still a considerable problem with resistance to change and the scaling up of good practice and innovation.If we take strategy and policy, there is an apparent mismatch between what is enshrined in assessment strategies and the reality of what is happening on the ground. These strategy documents tend to be quite procedural in focus and don’t reflect current thinking around effective assessment practice.Another key issue is that devolved responsibility for assessment and feedback across faculties and service departments results in considerable inconsistencies, making it difficult to achieve parity of experience for learners. In terms of infrastructure there has been considerable investment in enterprise solutions to support assessment and feedback but all too often the implementation of the technology has been seen as the solution with inadequate support for ongoing staff training and engagement. Institutional processes are also not always fit for purpose and the lack of interoperability between systems is a persistant problem.When it comes to assessment and feedback practice the issues are varied and complex but include the persistance of traditional forms of summative assessment such as essays/exams which increase the workload burden. Providing feedback is also an issue esp as deadlines are often tight and there is pressure on teaching staff to turn around feedback in tight timescales. Engaging learners can often be problematic as the value of acting on feedback is not always well-communicated. This goes back to the emphasis on high-stakes assessment and the value that is placed on marks and grades. Engaging with employers is causing many institutions to rethink approaches to assessment practice which often do reflect the reality of the professional world.
The technology landscape shows a wide array of technologies used to support assessment and feedback in a variety of ways, ranging from enterprise solutions to support formal processes to digital tools for teaching and learing. Most universities have this variety of technologies at their disposal but there is considerable variety in the extent to which the technologies are integrated or supported.On the whole universities are tending to work with technologies that are already well-established and will have a core group of technologies which they support but policies to support the use of external tools esp social media is variable and practice in the use of these is patchy.
So what approaches are institutions taking to address some of these issues? The examples I will talk about are around 2 main areas:Assessment practice which focuses on the educational aspects and learning processesAssessment management which focuses more on infrastructure and the institutional processes which support A&F and the student experienceA key themes which underpins these is the management of change - that is the effective strategies that drive forward and embed effective use of technology so I will come to this at the end.
The value of learners engaging with their feedback and the impact this has on improving performance has been strongly referenced in the academic literature. However, there is a perception that students don’t engage with feedback. There are various reasons why this might be but one factor which is born out by the National Student Survey is the timeliness and consistency/quality of feedback. Automation of the process could help alleviate the problem for example through audio feedback which research shows can be beneficial in terms of time-savings and student engagement.
To make feedback a more active experience for learners, the University of Westminster has developed the ‘e-Reflect’ process which utilises an online learning journal for student reflection on tutor feedback. The system enables feedback to become more active and personalised and joins up hitherto separate processes. Essentially the process enables a structured dialogue between the tutor and student and encourages the student to think more strategically about how they will act on feedback. Evidence suggest that this feedback process supported by technology is having a positive impact on learner engagement with feedback.There have been challenges with integrating the tool with other systems but the project is developing an LTI compliant version so that other institutions can operate the process within their VLE and benefit from the workflow element of the current tool.
Employability is a high-level priority for institutions. Influenced by the White Paper on Higher Education, Students at the Heart of the System, employability is increasingly challenging traditional approaches to assessment.Innovative practice in relation to assessment and employability includes increasing the focus on formative assessment to build the necessary skills and, wherever possible, introducing greater authenticity into assessment to improve the validity of the qualification for employers and learners.
A project at the University of Exeter aims to introduce new generation, 'authentic' assignments based around real-world and scenario based activities designed collaboratively by programme teams, students and employers. The project has developed a model to engage staff with principles around more employability-focused assessment design. It encourages teaching staff to think through more authentic work scenarios in their discipline area and devise assessments that develop skills around this eg. more frequent and short notice assignments, communicating to different audiences, collaboration and team work and peer-to-peer review and critique. To support this the project has developed a series of cards where technologies are ranked according to how they can support these different assessment contexts. This approach is interesting because the model and cards are evidence-based and aligned to educational principles in the context of work-integrated assessment. As a result it is much clearer for academics to see the potential of different tools to support the development of employability skills.
Having talked about some of the more educational issues and approaches to enhancing assessment and feedback, I’m going to turn to assessment management. By this I mean the management of the processes associated with assessment as well as the use of assessment data and how it can support decision-making around process change and the implementation of appropriate technology interventions. There is alot to say about learner analytics I’ll focus here on e-submission and touch on open approaches to assessment management.
Manchester Metropolitan University is undertaking a university-wide initiative to improve assessment and feedback practice. Their project is reviewing the entire institutional process for assessment management and developing systems to address each part of the lifecycle you see here. One key problem for the institution is is around assignment submission. 600,000 assessment marks need to be securely recorded for 36,000 students across a wide range of disciplines. Despite placing some constraints on the number of assessments so the total number has reduced, they still need to manage a huge variation in assessment practice across a large and diverse institution.
As part of the project, the university has used data to identify critical points for assignment submission (e or otherwise). The persistent emphasis on summative assessment towards the end of the teaching programme has implications for the workload of both students and staff as well as the supporting processes. To improve this some modelling has been done to identify significant peaks in assignment submissions (the highest being around 17,000 individual submissions due at the end of March 2012). This clearly shows where the main peaks are in the year so is helping inform what the best interventions might be to alleviate workload pressure.
Another project at the University of Huddersfield has been evaluating the benefits of e-submission and e-feedback. One key finding is that students feel more confident with the process. It’s faster, more efficient, convenient and flexible. But when it comes to receiving feedback, students see it as an inconsistency that this if often done in paper form when the assignment has been submitted electronically. It seems illogical to them that that inconsistency exists. There are also advantages to students engaging better with e-feedback and it providing benefits in terms of access for later use. Legibility of feedback is also a key advantage.
The technology transfer projects in the programme are not so much involved in organisational change but are showing how open approaches to assessment management can bring real benefits. The University of Nottingham together with 5 other universities has developed an open source, enterprise-level assessment management tool which can be used for both formative and summative assessment and feedback e.g. quizzes,exams, surveys. The tool has been designed around standards and supports QTI 1.2 import/export to its internal question bank, LTI support for integration with VLEs and LDAP for authentication.The project has shown the tool to be scalable, secure, reusable and is already in use in a number of partner institutions. The project is also working with partners in the Czech Republic and would like to build the community to beyond English speakers.Kingston University has been developing a tool with partners to make the authoring of e-assessment content more accessible and reusable. Again using open standards this makes QTI authoring more accessible to uninitiated users through a WYSIWYG editor and drag-and-drop facility making the management of assessment content more transferable, re-usable and user friendly. It’s worth mentioning that these project including a related project called QTI Works, have worked as a community to realise the QTI (Question & Test Interoperability) standard v2.1 to help drive forward open and standards-based assessment.
Enhancing assessment and feedback practice and the introduction of new ways of working across an institution involves a complex change process. Models and approaches to change across this Jisc programme of work have shown how stakeholders can be engaged effectively and that much of what surrounds successful change is winning hearts and minds more than anything else.
One of the key messages to reinforced by this programme is the importance of educational principles. The use of the REAP principles for assessment and feedback developed at the University of Strathclyde has gained traction across the sector and provides an educational scaffold on which to drive change aligned to educational principles. So it makes sense from the start to ensure that any technology-enhanced change is underpinned by some real understanding of the pedagogical basis for change. Once you have principles that align with your institutional contexts, you can look to drive change from there. This notion of principles can seems quite abstract but a number of projects are putting these into practice in a variety of ways to engage staff with the change process and encourage the appropriate use of technology to support learning.What has emerged is that principles provide a discourse for change and this can be a powerful lever to ensure that processes and practices supporting assessment and feedback all benefit learning.Hopefully this talk has given you a flavour of the types of challenges facing the UK sector which I’m sure are not uncommon beyond the UK and the ways in which institutions are driving forward change in assessment and feedback practice with technology.
