The document discusses improving course evaluation methods at universities. It notes low survey response rates are a key issue and that in-class administration can help increase participation. Students want feedback that directly benefits them and surveys should engage students and clearly communicate the actions taken in response. A consistent, centralized approach to surveys is recommended to allow benchmarking, but individual departments should have flexibility to include bespoke questions. Effective course evaluation provides evidence of a university's value to students.

1. Effective Course Evaluation
Summary report from interviews among
senior academics and student
representatives
 Improving response rates
 Improving student feedback
 Improving turnaround time
 Improving survey administration
 Improving the student experience

2. Effective Course Evaluation
Reality is:
There is a problem!
 Its 2012
 IT /process improvement have had a go
 Where do institutions go from here?

3. Improving response rates – from the report
 Many universities seeking feedback on courses and lecturers via
surveys struggle to achieve a meaningful response from students, which
is partly due to students being inundated with requests to complete
surveys.
 In-class student involvement in survey administration can increase
commitment as they are stakeholders in the process.
“A key issue facing universities in relation to gaining effective course
feedback from students is around achieving significant response rates
to any questionnaire-based approach. Students are increasingly
overloaded with surveys - from the National Student Survey downwards
- so you need a good reason to encourage participation.”
Professor Glenn Burgess, Pro-Vice-Chancellor of Learning and Teaching, University of Hull

4. Improving response rates – from the report
Group work:
Why is obtaining a high
response rate important?

5. Survey Best Practice rates – from our experience
Improving response
Is it a key challenge to achieve or maintain high
response rates?
81% Strongly agree
12% Neutral
6% Disagree
Population: 150 university administration staff at 70 institutions (UK)
5

6. Improving response rates – from our experience
Why is obtaining a high
response rate important?
Survey Sample  General student satisfaction survey
 Sample university student population
The University  Online survey
 20% response rate
 Draw conclusions

7. Improving response rates – from our experience
Why is obtaining a high
response rate important?
Survey Sample  Course/module/seminar evaluation
 10 – 30 students
The Course  20% response rate
 Statistically valid??
 Useful organisationally??

8. Survey Best Practice
In-class
Most likely to complete
Satisfied Dissatisfied
+30% higher response with in-class over email/VLE
online surveys
8

9. Survey Best Practice
Out of class
Most likely to complete
Satisfied Dissatisfied
“UK Students: The most distracted by social
media in the world” - Sept 2011 Telegraph
9

10. Improving student feedback – from the report
 Student representatives have indicated that students are not effectively
engaged in the feedback process and, for some, providing feedback can
even be intimidating.
 Universities need to work harder at feeding back to students the actions
they will be taking as a result of input provided for course and lecturer
evaluation surveys.
“Students need to provide feedback, then the universities need to take
that on board and do what they can with it, and then provide clear
feedback to the students on the actions and outcomes”
Alex Smith, former Academic Affairs Officer, University of Leicester Students‟ Union, and member of
Quality Assurance Agency Board of Directors.

11. Improving student feedback – from the report
Group work:
How would you improve
engagement in student feedback

12. Improving student feedback – from our experience
 Manage feedback „buy in‟ for all stakeholders
Student Lecturer School T&L Executive

13. Improving turnaround time – from the report
 Ideally students want the opportunity to express their views on course
improvements at a time that their feedback benefits them directly.
 End-of-module evaluation is a particular stumbling block in the provision
of feedback to students - and feedback can be slow - but moving to
mid-module evaluation can help to improve the process.
“Students should be able to express how their course could be
improved while it impacts on them.”
Alex Bols, Head of Education and Quality, National Union of Students

14. Improving turnaround time – from the report
Group work:
How would you improve turn
around time for module feedback

15. Improving turnaround time – from the report
What survey methodologies are used at your
institution?
9% Paper only
9% Online only
12% Hand-Key & Online
3% Hand tabulation
77% Paper & online
Population: 150 university administration staff at 70 institutions
15

16. Improving turnaround time – from our experience
Advantages
Instant feedback Low deployment costs
High response rate Easy for administration
Easy for participant Location independent
Onlin
Paper Lower risk Multi-delivery modes e
Delayed feedback
Higher deployment costs
Low response rate
More administration
More effort for
Sources of error
participant
Environmental concerns
Higher risk
Disadvantages

17. Improving turnaround time – from our experience
Advantages
Instant feedback Low deployment costs
High response rate Easy for administration
Easy for participant Location independent
Lower risk Multi-delivery modes Out of
In class
class
Delayed feedback
Higher deployment costs
Low response rate
More administration
More effort for
Sources of error
participant
Environmental concerns
Higher risk
Disadvantages

18. Improving survey administration– from the report
 Universities need to embrace new technologies to improve turnaround
time - but effective feedback can be gained via a combination of paper
and online surveys.
“There should be a unified approach that includes core questions and
specifies the way the feedback is processed - which allows
benchmarking and consistency. A common set of core questions should
be owned institutionally, with subject areas able to select from a bank of
optional additional questions.”
Professor Alan Speight, Pro-Vice-Chancellor (Student Experience and Academic Quality
Enhancement), University of Swansea, and member of Quality Assurance Agency Board of Directors.

19. Improving survey administration– from the report
Group work: (Module/Course level)
Academic led vs. centralised
module surveys – which is best?

20. Improving survey administration– from our experience
Who manages the survey process at your organisation?
50%
65% 65% Centrally
47% Department or
curriculum area
47% 50% Individual instructors
Population: 150 university administration staff at 70 institutions
20

21. Improving survey administration– from our experience
Who has the main responsibility for analysis and
reporting at your organisation?
15% 59% Centrally
59%
38% 35% Department or
curriculum area
38% Individual instructors
35%
15% Institutional
Research
Population: 150 university administration staff at 70 institutions
21

22. Efficiency and costadministration– from our experience
Improving survey savings
Individual Academic with 5 courses = 1 day per semester
(Survey Design: 2hrs / Deployment: 1hr / Data Capture: 1hr / Analysis: 4 hrs)
22

23. Efficiency and costadministration– from our experience
Improving survey savings
Department with 50 academics = 52 days per semester
(Individual survey admin: 50 days / Aggregate departmental analysis 2
days)
23

24. Efficiency and costadministration– from our experience
Improving survey savings
Arts Business Engineering
School
Law School Languages Sciences
Average University (academic run survey admin):
20 departments / 1000 instructors = 1040 days per semester
24

25. Efficiency and costadministration– from our experience
Improving survey savings
Best practice example: London School of Economics
23 Departments /1000+ staff / 9000 students
Complete survey administration 15 days per semester
(Deployment: 4 days / Data Capture: 10 days / Reporting: 1 day)
25

26. Improving the student experience – from the report
 Universities should establish a more consistent (centralised) approach
to survey administration - including a standard set of survey questions -
to enable effective benchmarking at course and institutional level.
However, individual departments should have the flexibility to include
bespoke questions for particular courses.
“With the National Student Survey, and the introduction of „Key
Information Sets‟, it is vital that there is an effective course and lecturer
feedback method so there can be constant improvements that will
determine a student‟s opinion of their institution based on direct
experience.”
Professor David Coates, Dean of School of Life Sciences‟ Learning and Teaching Division, University
of Dundee.

27. Improving the sutdent experience– from the report
Group work: (Module/Course level)
How can module evaluation
enhance the student experience

28. Improving the student experience – from our experience
 Centralise d survey unit (e.g. NSS, PTES, PRES, module etc)
 Common survey/evaluation data gathering (standardise format, reports
etc)
 Define a core set of questions for course evaluations
 Optional questions for departments but aim to use as few
questionnaires as possible
 Use course evaluation data as a means to improve the student
experience

29. Summary – from the report
“universities will be expected to publish online summary reports of student
surveys of lecture courses, aiding choice and stimulating competition
between the best academics”
Higher Education White Paper, June 2011
 Effective course evaluation is necessary for universities to provide a
clear evidence base to demonstrate their „value‟ to students.

30. Summary – from our experience
 Use in class time (paper or online)
 Target 50% response to maintain confidence and credibility of results
 Avoid incentivising students to complete surveys
 Avoid withholding grades
 Turnaround feedback to all stakeholders before the course finishes
 Tie results to action planning and improvements
 Gain stakeholder buy-in as the key to participate - „Myth of Survey
Fatigue‟
 Use „you said – we did‟ reporting
 Brand the process of giving feedback
 Ask students to evaluate their own effort on the course
 Reinforce the concept of anonymity to get honest feedback