1. Creatively engaged online:
student-generated content in a
non-majors introductory course
Emily Altiere, Simon Bates, Firas Moosvi
Department of Physics and Astronomy
University of British Columbia
AAPT Summer Meeting, July 2013, Portland OR
Monday, 15 July, 13

2. Outline
• Design and Methodology
• Engagement
• Question/Explanation Quality
Monday, 15 July, 13

3. a"web&based"MCQ"repository"created"by"students"
Monday, 15 July, 13

4. Ins$tu$ons((signing(up(per(year:(
!
2009:! ! !22(
2010:! ! !66(
2011:! ! !204(
2012:! ! !266(
2013!(Jan,Jun):! !214(
Growing(content(repository:(
!
Courses:! !2,500(
Logins/month:! !75,000(
Ques<ons:! !600,000(
Answers:! !12,000,000(
Monday, 15 July, 13

5. Student'ownership'over'
learning'resource'
Student'familiarity'with'
social'so7ware'
Leveraging'student'
energy'and'crea9vity'
Monday, 15 July, 13

6. Design and Methodology
Minimum participation requirements for each
of two assessment exercises (PW1, PW2)
Write 1 Answer 5 Rate / comment 3
5% course credit
Physics 101, Energy & Waves
Winter Semester: 3 sections, 791 students
Monday, 15 July, 13

7. Design and Methodology
TOCCLT'.'
-
QestIsct
How 1o...
q/t4osTer
Submit ond onswer questions
on topics in lhe torget region,
just obove lhe physics
you hove olreody mostered.
Dislroclers
) l*"*"ò
rNrrltng
Ih s region corrk:ins llte
plrysics knowledgo ond colrco¡ls
you connol leorn yel
becouse the louncalions
crrc nol in plocc
PHYSICS TOPICS IN YOUR TARG T REGION
buo'5ot't1 'Den¡¡lu
B¿:ovvì 1 J
F:rcq - YOUR CHOSEN TOPIC
à..
auC'q^ x@
bôo'osro
pn ro, d logroo,
o 9tS
COMMON MISCONCEPÎIONS AND ERRORS
(Sec hHÞ:/jÞhy¡.udofo¡.cdu/CJP/trêconcoÞllo¡t.pdt
tor o llil ol common mhconcepllons)
¿K N(,qht %cce Açxs no exi¡þ
, c.rrrçrJ On oblec'-t 5 õÞYJrnr cìü?À
I in rtuiò. ü
lrh'brot4ont eprç,e .I¡1enÀs crq
den-s,{^ c>Ç c.,þ¡ec , cìo+ .*¿n:r
oÇ Çtu,8,
fr- btrrqon! $crCe c¿cs Àoc¡:'cr
fr¿lt{e- r *hcrn ú?
Trr¿e. $ììfifflsz hrrr: ¿lÕoolr-g lnnS
(**= tcpo þ/rnl
{otr-* qrÇ bq.:. 0" Ooz
Physics knowledge
ond conccpluol underslonding
you lrove olrcody
conslrucleC ¡n your heod
¡,<'t.lerJ rt)^,¿z! Þ.rrp r- rr't¡ ,l?
Chcck lhol yovr on3Íêr ¡t rêo!ànoblê ðnd potrlble
'f<,,cc
Photo by Seth Casteel
http://www.littlefriendsphoto.com
Permission to use agreed
Introduced in tutorials
Extensive scaffolding
exercises
Revisited in subsequent
tutorials
Tutorials delivered by 24 TAs
Monday, 15 July, 13

8. Engagement with PeerWise
Monday, 15 July, 13

9. Engagement with PeerWise
PW1 PW2
Contributed and met minimum requirements
Contributed but did not meet mininum requirements
Did not participate
0.84 0.80
Total: 791
Monday, 15 July, 13

10. Engagement with PeerWise
Number Multiplier Number Multiplier
Questions 1105 [1.7] (1.4) 998 [1.6] (1.3)
Answers 11393 [17.2] (14.4) 11807 [18.7] (15.0)
Comment
s
4901 [7.4] (6.2) 5509 [8.7] (5.0)
PW 1PW 1 PW 2PW 2
Monday, 15 July, 13

11. Question/Explanation Quality
Monday, 15 July, 13

12. Question/Explanation Quality
Bloom’s Taxonomy of levels in the cognitive domain
Score Level Description
1 Remember Factual knowledge, trivial plugging in of numbers
2 Understand Basic understanding of content
3 Apply Implement, calculate / determine. Typically one-stage problem
4 Analyze Typical multi-step problem; requires identification of strategy
Evaluate Compare &assess various option possibilities; often conceptual
Synthesize
Ideas and topics from disparate course sections combined.
Significantly challenging problem.
Monday, 15 July, 13

13. 1 2 3 4 1 2 3 4
0
20
40
60
Numberofquestions
Assessment 1 Assessment 2
Bloom's Taxonomy: Question Quality
Textp>0.05, NS
Monday, 15 July, 13

14. Question/Explanation Quality
Description of explanation quality
Score Level Description
0 Missing
No explanation provided or explanation incoherent/
irrelevant
1 Inadequate Wrong reasoning and/or answer; trivial or flippant
2 Minimal
Correct answer but with insufficient explanation/
justification/ Some aspects may be unclear/incorrect/
confused.
3 Good
Clear and detailed exposition of correct method &
answer.
4 Excellent
Thorough description of relevant physics and solution
strategy. Plausibility of all answers considered. Beyond
normal expectation for a correct solution
Monday, 15 July, 13

15. 0 1 2 3 4 0 1 2 3 4
0
20
40
60
Numberofquestions
Assessment 1 Assessment 2
Explanation Quality
p=0.02
Monday, 15 July, 13

16. Conclusions
Participation is generally well beyond the minimum
requirements for course credit.
Unlike similar studies with Physics majors*, non-majors
in this study produced questions of lower overall
quality.
Evidence to suggest that question quality and detail
of explanation improves with practice.
*Bates, et al ‘Assessing the quality of a student-generated question repository’ PRST-PER submitted
Monday, 15 July, 13

17. References
Student-facing system
http://peerwise.cs.auckland.ac.nz/
All the research studies referenced and scaffolding materials
referred to are accessible through the PeerWise community
site http://www.peerwise-community.org/
Monday, 15 July, 13