Part 2 of Paul Denny's presentation at the LTKB workshop, Edinburgh 2011. PeerWise is a web-based repository of MCQs built by students. Students are given the responsibility of creating and moderating the resource. By leveraging the creativity and energy of a class, a large, diverse and rich resource can result.

1. University of Edinburgh
Peer Feedback and Assessment for Science and Engineering
17th December, 2010
PeerWise
familiarity breeds content
student-authored questions using PeerWise
Paul Denny
Department of Computer Science
The University of Auckland
New Zealand

2. Goals
• Hands-on with PeerWise
– authoring, answering, evaluating
• Selected results
• Getting started
– with your students

4. What is PeerWise?
• Web-based MCQ repository built by students

5. Your turn
Google: peerwise
or
http://peerwise.cs.auckland.ac.nz

6. Registration
Click the “Registration” link

7. Registration
Begin registration

8. Choose a user name
Enter any user name you like

9. Choose a password
Choose a password for
your account. Make sure
you remember what this
is!

10. Enter the “Course ID”
The Course ID for this
workshop is:
5279
Enter this Course ID

11. Enter your “Identifier”
Enter your “Identifier” value

12. Confirm
You should see a green tick. Just click
“Confirm”

13. You can now log in and begin...
You can now log in with
your username and
password.

14. Hands-on demonstration
• Creating, answering, evaluating questions

15. Research
• What do students think?
– Auckland
– Edinburgh
• Repository quality?
• Activity linked to performance?

16. What do students think?
“I actually found it enjoyable, as sad
as that sounds.”
Student feedback

17. What do students think?
• Survey responses (n = 439)
– ENGGEN 131, Semester 2, 2007
Developing new questions Answering other student's
helped me learn questions helped me learn

18. What do students think?
• Survey responses (n = 387)
– ENGGEN 131, Semester 2, 2009
Developing new questions Answering other student's
helped me learn questions helped me learn

19. What do students think?
• Focuses attention on learning outcomes
"What I found most interesting was how
other people structured their questions.
It kind of made me think about what kind
of topics or concepts people felt were
important to the course, hence their
choice of making a question on a
particular topic."

20. What do students think?
• Focuses attention on learning outcomes
• Express understanding in their own words
"The biggest learning experience for me was
setting up my multi-choice question...
...in the end it was a lot of help because i was just
about able to answer any question that was on
the same topic as my question"

21. What do students think?
• Focuses attention on learning outcomes
• Express understanding in their own words
• Question bank for drill and practice revision
"I answered over 100 questions, it was a
quick way to test my knowledge and if I
got the answer wrong the explanations
helped me learn something I wasn't too
sure with."

22. What do students think?
• Focuses attention on learning outcomes
• Express understanding in their own words
• Question bank for drill and practice revision
• Peer comparison
"Being able to see how other people answered was
great as it allowed me to recognise at which level I
was at compared to everyone else"

23. Edinburgh
• We sought student feedback both in ‘wash-
up’ sessions after the assessment and in the
end of course questionnaire

24. Edinburgh

27. Repository quality?
“The quality of questions range from
meaningless questions to questions
that stimulate your brain.”
Student feedback

28. Repository quality?
• Selected finding
– how often is the author’s answer incorrect?

29. Repository quality?
• Selected finding
– how often is the author’s answer incorrect?
COMPSCI 101, Semester 1, 2008 BIOCHEM 233, Semester 1, 2010
617 questions 528 questions
Analysis: 10% (62 questions) Analysis: 20% (109 questions)

30. Repository quality?
• Selected finding
– how often is the author’s answer incorrect?
COMPSCI 101, Semester 1, 2008 BIOCHEM 233, Semester 1, 2010
617 questions 528 questions
Analysis: 10% (62 questions) Analysis: 20% (109 questions)

31. At OzBio 2010, Melbourne, October 2010
Dr. Steven Bottomley
Faculty of Health Sciences, Curtin University

33. Edinburgh
• Quality of submissions:
– Average quality was very good
– Few trivial questions / nonsense distracters
– Highest quality questions were EXCEPTIONALLY
good

34. Activity linked to performance?
“Using PeerWise was very useful for
learning things I didn't know I didn't
know.”
Student feedback

35. Activity linked to performance?
• Do the most active students improve their
position in class?
– study at the University of California, San Diego
prerequisite
CSE 8A CSE 8B
Fall 2008 Winter 2009

36. Activity linked to performance?
• Do the most active students improve their
position in class?
– study at the University of California, San Diego
CSE 8A CSE 8B n = 73
Fall 2008 Winter 2009
Final exam grade provides Final exam grade provides
class rank in CSE 8A class rank in CSE 8B

37. Activity linked to performance?
• Quartiles based on number of questions
answered
Q1 Q2 Q3 Q4

38. Activity linked to performance?
• Change in rank from CSE 8A to CSE 8B
– (mean number of questions answered)

39. Activity linked to performance?
• Replicated the following term
Winter 2009 (n=73) Spring 2009 (n=53)
The highest performing The lowest performing
students in the pre-requisite students in the pre-requisite
course were most active course were most active

40. Future work
• Multi-institutional collaboration
• Challenges
– Calendars
PeerWise
– Curriculums
Course 1 Course 3
Institution A Institution C
Course 2
Institution B

41. Future work
• At Edinburgh
– Does use improve course performance?
– Enhancements / changes for next year
• More consistent use across course
• More guidance from TAs / Staff?

42. Creating a new PeerWise course

43. Creating a new PeerWise course
123 234 345 456 567

44. Creating a new PeerWise course
• Step 1)
• Step 2)

45. Thank you
• Any questions?
– Now
– Later
Paul Denny
paul@cs.auckland.ac.nz
Department of Computer Science
The University of Auckland
PeerWise
peerwise.cs.auckland.ac.nz