Fitzpatrick, M.; Risquez, A. (2012): The invisible student – the visible feedback: The impact of lecture attendance and online administration on Student Evaluation of Teaching (SET) Feedback. ISFAM 2012, 26th - 29th June 2012. University of Limerick, Ireland. Kemmy Business School.

1. The invisible student – the visible feedback:
The impact of lecture attendance and online
administration on Student Evaluation of
Teaching (SET) Feedback
Dr Mary Fitzpatrick
&
Dr Angelica Risquez
Centre for Teaching and Learning,
University of Limerick

2. Attendance crisis in Universities?
How many of you
experience
attendance issues in
your classes?
Student make informed choices in terms of not
attending class – this is the new reality

3. Student Online SET
feedback (student volunteers,
lower response rates
and other factors)
- Student
commitment and
interest in subject Level of Feedback
attendance? scores
- Student level of
engagement
-Performance

4. Factors impacting on SET
• SET ratings often influenced by factors outside
of the teaching effectiveness scale (Hackett,
2006)
• These impact on the level of student
engagement and attendance to class
• Should these invisible students (who have not
attended) provide feedback on the entirety of
the module?

5. How to read SET?
• Students don’t tend to blame themselves for
their problems with a module or the fact that
they are not attending – they attribute this to
their teachers and the actual module
(Davidovitch & Doen, 2006)
• Academics are left reeling with evaluations
despite the fact that many of the students
have not attended

6. The Dilemma
• How can we rely on validity of SET
systems if they are completed by
absent students?

7. SET and the captive audience
Online Vs In class evaluations

8. Paper – in class Online
• Cumbersome • Flexibility in administration
• Longer turnaround times • Rapid archiving of data
– constrains the ability of • Minimise demand on
teaching staff to take on students and class-time
board the feedback • Eliminates paper and the
during the semester or time-consuming optical
to use the data scanning process and paper
effectively handling
• Improves turnaround time
(Kiesler & Sproull, 1986)

9. Method of administration – impact
on results?
Potential Lower response
negative bias; rates but minimal
Convenient but impact on results
unreliable

10. Need for further study because…
• These studies refer to small and cross
sectional samples
• This is a longitudinal study based on a very
large sample

11. Research methodology
(Controlling for other contextual variables) it
is proposed that…
class attendance and online SET
administration do not impact on
average SET results

12. Survey Instrument
• Survey with 10 Lecturer items, 8 Module items and 7
Student items, 5 point Likert scale
• Item “I attended most or all of the required contact
hours for this module” to infer attendance
• We use average scores in the Lecturer and Module
scales
• Multivariate regression analysis
SET average score = attendance * paper Vs online * other
controlling factors

13. Sample
• 88,686 student responses collected over a period of
11 years
• All evaluations requested by their lecturer
• Representation
– 83% in the traditional age group (18-23)
– 87% undergraduates
– 42% Business, 28% Science and Engineering, 17%
Humanities and 12% Education & Health Science
– 36% first year students, 22% second year, 19% third year
and 22% final year

14. Findings and results
Variables R2 Sig. Beta Coefficients
controlled
Lecturer Faculty of S&E, 0.102 p=<.001 Attendance=0.182
scale class size, years Paper admin=0.086
average teaching
Module Faculty of S&E, 0.098 p=<.001 Attendance=0.24
average class size, years Paper admin=0.066
scale teaching, year of
evaluation

15. Findings and results
• Lecturer Scale: attendance accounts for an
increase of 0.182 in average scores, paper SET
administration accounts for an increase of
0.086 in average scores
• Module Scale: attendance accounts for an
increase of 0.24 in average scores, but paper
SET administration accounts for an increase of
0.066 in average scores

16. Class attendance does have some
(positive) impact over SET scores
• On its own, class • The combination of
attendance accounts class size, faculty, year
for 4% of the variance of evaluation, years of
of scores in the lecturer teaching experience,
scale averages and attendance and method
almost 6% of the of administration can
module scale averages explain up to 10%
variance of SET scores
in lecturer and module
items

17. But online administration is not such
a devil…
• (In line with previous findings)
online administration has not
been shown to have such a
decisive impact on SET scores

18. Limitations
• Results must be generalised with care
• Sampling method is an issue both in paper
based and online SET administration –
attendance is still an issue!

19. Implications
• “Invisible” student provides (slightly) less
positive feedback
– why is this the case when they don’t attend the
classes?
• SET offers many possibilities to access the
thoughts of students who seem to be falling
though the cracks – certainly worth
considering…..

20. Thanks!
• Any questions???
– Dr. Mary Fitzpatrick, mary.fitzpatrick@ul.ie
– Dr. Angelica Risquez, angelica.risquez@ul.ie