5. So, Can We Provide
Incentives,
and will it change behavior?
6. What Happened?
Question 1 (Cross Sectional Analysis):
Did instructors who used incentives have higher student response rates
than instructors who did not?
• Mean rate without incentives: 47.3% of students completed part or all of
on-line ClassEval
• Mean rate with incentives: 79.6% of students completed part or all of on-
line ClassEval
• This difference was significant by both
- t-test: t (df = 44) = 7.54, p < .0001 (percentages require arcsin
transform)
- Non-parametric test (Krukall-Wallis) p< .0001
• So, instructors that used class-based incentives had dramatically higher
student participation in ClassEval
-- But maybe those instructors would have had higher participation
rates anyway
7. What Happened?
Question 2 (LongitudinalAnalysis):
Did instructors who used incentives in Spring ‘10 have higher
evaluations than for same course in Spring ‘09 (when no one used
incentives)?
(Note: Not all instructors taught the same course in both ‘09 and ‘10)
• So, even the same instructors had higher student participation when
they used class-based incentives
% response in
Spring ‘09
% response in
Spring ‘10
No incentives in
Spring ‘10
49.4% 46.6%
Used incentives
in Spring ‘10
59.0% 77.6%
8. What Happened?
Question 3 (one more interesting question):
Do evaluations scores correlate with the response rates?
• Here is the scary part – without incentives the fewer the students
who participate, the lower the mean ratings of the class
• But, if you use incentives, the number of students participating has
no relationship with mean ratings of class
• Why?
Did not use
Incentives
Used Incentives
Q. 9 (Instructor) .39 -.01
Q. 14 (Course) .49 .00
Students evaluate their classes from very early on. In Alaska, this student evaluation would not be counted.
As a psychologist, I believe in using incentives to get people to behave in desired ways
Telling students to do evaluations for the common good may not be enough incentive. This might be called the “Broccoli” approach. Broccoli may be better for future Doug, but present Doug may find the bacon cheeseburger to be more rewarding to the sensory system.
The frontal and prefrontal cortex plays an important role in decision making. They aren’t well developed in college students. So thinking about the future is not as likely.
In my department last spring, we did a naturalistic experiment – I suggested that instructors might try a class-based incentive, such as if 80% of the class does the on-line student evaluation, then the final exam will include an extra credit question. About half of the instructors used class-based incentives and half did not (they used exhortation to do the right thing)
Any class has a number of dissatisfied students – they have an incentive to respond to the ClassEval survey. So, their numbers stay about the same. With low response rates, the evaluations will be sensitive to the number of the reasonably satisfied students that participate. But once you have a larger number of respondents, the weight of the dissatisfied students doesn’t vary much as the overall number of respondents changes.
Thinking that students will participate in ClassEval without incentives is the stupidest idea in the history of stupid ideas. It is so stupid that I suspect the real intent was to undermine the system, to make sure that it failed. The only problem is that ClassEval data are used for promotion and tenure, for yearly evaluations, by faculty to revise classes. So now we are stuck making all of those decisions using unrepresentative or bad data.