2. Scene setting
Student population
– Undergraduate 19,096
– Postgraduate Taught 5,147
4 Colleges
– Arts
– Medicine, Veterinary and Life Sciences
– Science and Engineering
– Social Sciences
Total of 19 Schools and 9 Research Institutes
3. Course evaluation
Session 2012-13 - EvaSys piloted
– 1 School per College
Working Group established to review
Recommendations:
• EvaSys rolled out on a voluntary basis
• Review of course evaluation
4. Issues
• Not compulsory to use questionnaires for
course evaluation
• Previously introduced standard questionnaire
– lasted one year!
• Large and varied means of evaluating
courses
• When questionnaires were being used – a
wide assortment of areas and questions
were being asked.
5. Just what are we evaluating?
What do we use course evaluation to evaluate?
6. Areas evaluated
• The relevance of the course?
• The student experience on the course?
• The teaching content?
• The quality of teaching?
• The assessment methods?
• The ‘popularity’ factor?
Then, there is…..
7. Areas evaluated
• Assessment of dissertation/project
supervisors
• Assessment of placements
• Assessment of learning and teaching
support
• Assessment of accommodation
• Assessment of laboratories, seminars
• Assessment of GTAs
• Assessment of feedback received
8. What do we do w
What do we do with the information?
• Management overview/auditing purposes
• Provide feedback to lecturers
• Provide feedback to course organisers and
programme directors
• Provide feedback to Head of School, Head of
College
• Provide information for marketing purposes
• Provide feedback to students!
9. Working Group on course feedback questionnaires
• To identify the purpose of course evaluation
• The nature of feedback processes
• Consideration of the use of questionnaire
data
• Administration of feedback processes
10. Summative
• To provide a snap-shot of the past, a
description of how the course ran
• To highlight existing good practice
• To highlight exceptional quality that can be
referred to in marketing and recruitment
activities
• To recognise teaching as evidence for
teaching excellence awards or promotion.
11. Formative
• To help identify where improvements can be made,
and where there are unexpected problems that need
to be solved
• To ensure that the quality of teaching is sustained
and improved
• To identify where teaching staff might need
additional support or resources
• To demonstrate the use of feedback practices
across the university for the purposes of audit
13. Course feedback evaluation questionnaire
• 5 core questions to be used on all
questionnaires
• Teaching and course evaluation to be
reviewed separately
• Optional question sets to survey different
aspects
14. Core Question Set
CORE1a. The lecturer explained things well.
• Strongly Agree □ □ □ □ □ Strongly Disagree
• or
• CORE1b. My project/dissertation/placement supervisor was helpful.
• Strongly Agree □ □ □ □ □ Strongly Disagree
•
CORE2. The course was intellectually stimulating.
• Strongly Agree □ □ □ □ □ Strongly Disagree
•
CORE3. I am satisfied with the overall quality of the course.
• Strongly Agree □ □ □ □ □ Strongly Disagree
•
CORE4. What was good about the course?
• OPEN
•
CORE5. How could this course be improved?
• OPEN
15. Clarify what is being evaluated: Other Question sets
• Teaching Quality set (9 qns)
• Course Quality set (9 + 5)
• PGT set (4)
• Expectation/Marketing set (5)
• Ad hoc set (up to max 22 qns)
16. What to do with the information?
• Course
– Improving course design
– Involving students with curriculum design and
development
• Teaching
– Improving teaching
– Provide source for recognition and award of
teaching
• Feedback
– Improving student feedback
17. Linkage to other processes
• P&DR and Recognition of Excellent
Teachers
• Annual Monitoring
• Internal Subject Review
• Marketing
18. Moving Forward
Establishment of Advisory Board
– Strategic overview
– Review potential developments and linkages to
other processes
– Identification and dissemination of good practice
– Update and revise policy as required
Large, diverse institution – restructured in 2010 – moving from 8 Faculties to 4 Colleges. Each College having a substantial level of autonomy
Voluntary basis? As an registration system had been recently introduced that had not gone smoothly – academics would not have warmly received another “imposed” system
As the Working Group reviewed the software –the benefits of EvaSys were recognised but for it to be effective – required a level of standardisation across the University for cross-Institution comparison for Quality Enhancement activities.
This was currently not standard practice at Glasgow
Academics found the standard questionnaire too structured - considered unsuitable and inflexible.
Working Group identified a large range of methods for course evaluation – focus groups, student interviews and question/answer sessions at end of class popular ways for obtaining feedback
And when questionnaires where being used – wide range of questions were being used.
So – Working Group recommended that both the role of course evaluation as well as using course evaluation questionnaires should be re-examined.
Seems like a very basic and stupid question but what is evaluated as part of course evaluation?
Some of these look similar, but actually are quite different…
Quite a lot going on – and often feedback was being sought on all of the above!
And What where we doing with the information collated?
Supposed to be doing – but were we really? So another Working Group was established – Course Feedback Questionnaires
It concluded that the purpose of course evaluation was -
Important to mention WG convened by Dr Helen Purchase (Computing Science) (and the EvaSys Working Group chaired by Dr Moira Fischbacher-Smith (Dean (L&T)) Invested substantial time in the group and establishing the report and final recommendations)
So what did the Group conclude was the purpose of course feedback?
But how do we achieve this from a questionnaire and what questions should be asked?
The Group reviewed 300 questions
Questions were placed into groups - which highlighted the different ways to ask the same thing! Often similar questions included on the same questionnaire, some misleading, some confusing and open for misinterpretation
Not only looked at questions – but design of questionnaires (eg The appropriate number of tick boxes? (we went with 5!) Good Left or right? Psychology of completing questionnaires, the ideal length, etc
Decided: what was the optimal number of questions?
Ask group for ideal number of questions….
THEN…ask them to go into groups and discuss what they would think the 5 questions the Group came up with
Once discussion has taken place ……
Out of 300 questions – agreed on 5 core questions! These are the only ones that are compulsory.
In a group – take 10 minutes to discuss what questions you would consider core?
It was agreed that this met all requirements -
This was the core information required to fulfil the needs of the University, the course organiser, the lecturer, the student.
3 scaled; 2 open questions
The 5 questions would be the only compulsory element to the questionnaire. Schools could choose to use just these 5 questions.
However, if more detail was required….
Question sets were identified to target specific areas
Remember all those 300 questions – went through to find ideal questions for each set.
9 questions for Teaching set
9 + 5 for Course set
4 for PGT set
5 for Marketing set
Ad hoc – any other questions the School wished to add
Maximum number of questions – should not exceed 22
Okay – now we had a template for our questionnaires - often next stage of evaluation can go astray – what do we do with the information collated?
Next stage – establishing viable systems are in place to ensures outcome of course evaluation are effectively managed and used.
Feedback on courses should fed into course design, should involve students
Feedback on teaching should improve teaching and be recognised as part of the recognition and award schemes
Students should be aware that their feedback is being taken on board
Also – potential to link into other processes
Anything else?
Next stage:
Now we have our questionnaire, we know what we want to do with our feedback – now need processes to ensure successful transfer of information and communication -
Advisory Board recently created to oversee EvaSys implementation, updating and revising policy as required
Oversee evaluation processes are working and enhancing other processes
Identification and dissemination of good practice
Identification of appropriate support and training
And finally …. To oversee future developments
This is the stage where we are currently at – Board is convened by the same person who chaired the Working Group – for continuality – representation from all Colleges both academic and administrative and from IT Services.
Might have to be a future event – where we report back whether or not our strategy has been successful!
Any questions?