Presented at the CyberSTEM conference, Berekeley, CA, 2011

1. Cyberlearning Tools for STEM
Education Conference 2011
Join the online discussion
backchannel for this session at:
www.cyberlearningSTEM.org/sessions

2. SCable
Real-time
SCRIBL Data
Justin Reich
Individual Harvard Graduate School of Education
Behavior and Cyberlearning Tools
Learning for STEM conference
3/8/11
Data bjr795@mail.harvard.edu

6. MrBoyersClass.Pbworks.com
Page Saves by Day
100
90
80
70
60
50
Page Saves
40
30
20
10
0
0 50 100 150 200 250 300
Days

7. Scatter Plot of Wiki Page Saves by Day
(n=1,799)

8. Moving average of wiki development
measured in page saves
ALLPS
30
20
10
0
0 100 200 300 400 500 600 700 800 900
day

10. SCable
Real-time
Individual
Behavior and
SCRIBL Data
Learning
Data

11. # of Time/Scale Web 2.0 Research
Cases
State Space Modeling
1,000K
Usage
Simulations Statistics
100K
Policy
Semantic
Makers
Analysis
10K
Surveys
1,000
School Content
Interviews Leaders Analysis
100
Discursive
10
Analysis
Teachers
Biometric Design Observational
1
Analysis Research Research
Seconds Days Weeks Months Years
11
Duration of data collection and capture

12. Research with SCRIBL Data
• We can study SCRIBL data with
microscopes and telescopes
– Plan for interdisciplinarity
– New methods are needed
• Operationalize time
• Don’t invent new platforms, meter widely
adopted platforms

14. Learner Analytic Approaches
Netflix vs.Pandora

15. PBworks wiki lifetimes in seconds/days
1
0.9
(n=179,851)
Estimated Survival Probability
0.8 Estimated Seconds Days
0.7 Lifetimes
25% 250 <1
0.6
50% 123,613 1.4
0.5 75% 5,282,874 61.1
0.4
0.3
0.2 All PBWorks Wikis
0.1
0
0 20000000 40000000 60000000 80000000 10000000
(231) (463) (694) (926) (1157)
Time in seconds (days)

16. Learner Analytic Approaches
Netflix vs.Pandora

17. Does wiki persistence differ in Title I and non-Title I schools?
(n=259)
Non-Title I Eligible
Title I Eligible

18. Wiki Opportunities for Students to Develop
21st Century Skills
• Expert thinking:
– Do students use academic content knowledge in wiki activities?
– Do students reflect on the process/product?
• Collaboration:
– Do students concatenate text on pages?
– Do they substantively edit each others work and co-create
pages?
• New Media Literacy: Wiki Quality Instrument
– Do students use formatting? 25 Questions
– Do they hyperlink?
– Do they embed multimedia?
Scale of 1-25

19. Do wikis provide opportunities for students to
develop 21st century skills?
25 Are great wikis born or made?
20
Wiki Quality Score
15
10
5
0
0 50 100 150 200 250 300 350 400 450
Days

20. Do wikis created in high SES schools provide more
opportunities for 21st century skill development?
25
20
15
10
5 High SES
Low SES
0
0 50 100 150 200 Days 250 300 350 400 450

21. Do wikis created in different subject areas provide different
levels of opportunity to develop 21st C skills
25
20
Wiki Quality Score
Social Studies
English
15
Science
Computer Science
10 Math
5
0
0 50 100 150 200 250 300 350 400 450
Days

22. SCable
Real-time
Individual
Behavior and
SCRIBL Data
Learning
Data

23. Educators routinely try to gather information
about their students’ learning on the basis of
what students do in class. But for any question
posed in the classroom, only a few students
respond. Educators’ insight into what the
remaining students do and do not understand is
informed only by selected students’ facial
expressions of interest, boredom, or puzzlement.
To solve this problem, a number of groups are
exploring the use of various technologies to
―instrument‖ the classroom in an attempt to find
out what students are thinking.

24. Back Deck

25. Classroom Wiki Research Questions
• How do we make them good?
– What best practices, attitudes and resources
produce wiki learning environments that
promote and nurture 21st century skills?
• Do only certain kids get the good ones?
– What is the distribution of high quality wikis
across schools serving different student
populations?
25

26. Open Education Resources
• Twin Hopes for OER
– (Excellence) Teachers will use free, online
tools and resources to create student-
centered learning environments that prepare
students for 21st century life
– (Equity) Since these materials are free, poor
students will disproportionately benefit.
Brown, J. S., & Adler, R. P. (2008). Minds on fire: Open education, the long tail, and learning 2.0. Educause
Review, 43(1), 16-32.
Bonk, C. J. (2009). The world is open : How web technology is revolutionizing education (1st ed.). San
Francisco, Calif.: Jossey-Bass.

27. Narrowing gaps or a rising tide?
Non-poor students Non-poor students
21st Century Skills
Poor students
Poor students

28. Remapping the Digital Divide
• How can we design a study to examine
whether or not poor students
disproportionately benefit from the
availability of free online tools?
– Theoretical Framework
– Operationalizing the theoretical framework
– Research Design

29. Dimensions of the Digital Divide
1st Digital Divide: Access 2nd Digital Divide: Usage
• Schools • Attewell (2003): ―[There exists a]
– 3.8 Students/Computer in real possibility that computing for
schools with <35% already-disadvantaged children
students eligible for FRPL may be dominated by games at
home and unsupervised drill-
– 4.0 students/computer in
and-practice or games at
schools with >75% student
school, while affluent children
eligible for FRPL
enjoy educationally richer fare
• Anywhere with more adult involvement‖
– 86 % of students living in • Jenkins (2007) Participation Gap
households making <$30K
Attewell, P. (2003). Beyond the digital divide. In P. Attewell, & N.
use the internet Seel (Eds.), Disadvantaged teens and computer
technologies (pp. 15-34). Munster, Germany: Waxmann.
– 97% of students living in Jenkins, H.; Clinton K., Purushotma R., Robison A. and Weigel
M.(2007), Confronting the challenges of participatory culture:
households making >$70K Media education for the 21st century. Chicago, Il.:
MacArthur Foundation.
use the internet

30. How can we operationalize usage?
Usage as persistence
• Wiki lifetime: number of days of activity of
a wiki community
– Birth: Creation of wiki subdomain (e.g.
ReichWorldHistory.pbworks.com)
– Death: Final wiki edit
• After a 90 day observational period
30

31. How can we operationalize usage?
Usage as opportunities to develop 21st
century skills
• Participation
• Expert Thinking
• Complex Communication
• New Media Literacy
31

32. Research Design

33. What are our research questions?
RQ #1) Persistence: Are wikis created in
schools serving affluent students used for
greater lengths of time than wikis created
in poor schools?
RQ#2) Participation: Do wikis created in
schools serving affluent students provide
more opportunities for students to develop
21st century skills
33

34. Which wikis are in my sample?
• Dataset
– All179,853 publicly-viewable education-related wikis
started on the PBworks platform between June 2005
and August of 2008.
– Does not include ―private‖ wikis (~70,000)
• Sample
– Randomly sampled 1,799 wikis (1%)
– Coded to identify 259 U.S. based, K-12 wikis from
specific public schools
• Detailed usage statistics provided by PBworks.com
• Demographic school level data from the Common Core of Data
(National Center for Education Statistics, 2007-2008) 34

35. What are our data analytic
strategies?

36. RQ#1) Are wikis created in non-poor schools used for
greater lengths of time than wikis created in poor schools?
• Estimate survival functions of wiki groups by Title I status
using Kaplan-Meier estimation; use Wilcoxon’s test to
test for differences
where
S(ti) is the estimated survival probability in any of t time
periods, which are delineated by instances of wiki death
ni is the number of wikis still active at the beginning of time
period ti;
di is the number of wikis that become inert during time
period ti.

37. RQ#2) Do wikis created in non-poor schools exhibit
more evidence of collaboration and student
involvement than wikis created in poor schools?
Estimate wiki quality trajectories using the multilevel
model for change.

38. Findings

39. How long do K-12 wikis persist?
(n=411)
Estimated Seconds Days
1 Lifetimes
All PBworks
0.9
25% 250 <1
0.8 50% 123,613 1.4
Survival Probability
0.7 75% 5,282,874 61.1
0.6 K-12 Wikis
25% 2,721 <1
0.5
50% 763,195 8.8
0.4 75% 12,590,074 145.7
0.3
All PBWorks Wikis
0.2
K-12 Wikis
0.1
0
0 20000000 40000000 60000000 80000000 10000000
(231) (463) (694) (926) (1157)
Time in seconds (days)

40. What subjects are wikis used for? (n=411)
English / Language Arts 120
Social Studies 70
Science 61
Computer Science/ Technology 60
Math 45
Library 26
Art 22
Contained Elementary 20
Modern FL 10
Health/PE 8
Business 6
ESL 5
Classics 4
Education 2
0 20 40 60 80 100 120 140

41. Does subject area predict persistence? (n=411)
50% 75% Lifetime
Lifetime
Subject
Computer Science 28 198
English/Language Arts 27 198
Science 18 152
Social Studies 6 56
Math 4 33
No Subject 1 8

42. What Grade Levels are K-12 wikis used in?
(n=411)
K-5 109
6--8 118
9--12 180
Higher Ed 8
Unknown 83
0 50 100 150 200

43. Does grade level predict
persistence? (n=411)
• No

44. Do wikis from non-poor schools persist longer than
wikis from poor schools? (n=259)
Kaplan Meier Survival Estimates of Wikis from Poor and Non-Poor Schools
Non-poor schools (n=146)
Poor schools (n=110)
Days 44

45. Summary statistics of wiki lifetimes in poor
and non-poor schools (n=259)
• Day 1 Mortality:
– Wikis from poor schools: 40%
– Wikis from non-poor schools: ~20%
• Median Lifetime:
– Wikis from poor schools: 7 days
– Wikis from non-poor schools: 58 days
• 25% Lifetime:
– Wikis from poor schools: 73 days
– Wikis from non-poor schools: 259 days 45

46. RQ#2) Do wikis created in non-poor schools exhibit
more evidence of collaboration and student
involvement than wikis created in poor schools?

47. Title I eligible Non-Title I eligible
(n=110) (n=146)
Concatenation 12 24
Copyediting 10 15
Co-construction 7 7
Commenting 16 22
Collaborative Sum= 0 82 (75%) 110 (75%)
Collaborative Sum= 1 17 (15%) 13 (9%)
Collaborative Sum= 2 7 (6%) 17 (12%)
Collaborative Sum= 3 2 (2%) 3 (2%)
Collaborative Sum= 4 2 (2%) 3 (2%)
2 Goodness of fit test ( 2=4.2, df=4, p=.38)
Student Involvement 28 (25%) 50 (34%)
2 Goodness of fit test ( 2=2.28, df=1, p=.13)
Student Involvement and at least 1 16 (15%) 20 (14%)
Collaborative Behavior
2 Goodness of fit test ( 2=0.04, df=1, p=.85)

48. Discussion
• Participatory behavior is rare across all wikis, but
both student involvement and collaboration can
be found in wikis from both poor and non-poor
schools
• Wikis from non-poor schools persist longer than
wikis from poor schools.
• The Open Education Resources strategy of
promoting free online tools and resources may,
counter-intuitively, expand the second digital
divide—in the absence of targeted interventions.
48

49. How can we begin to explain
these patterns?
• What might explain our empirical findings?
– What obstacles do poor schools have in using wikis?
• Qualitative Research
– Interviews with 50+ wiki-using teachers, many drawn
at random from our PBworks samples.
– 35+ focus groups with students
– Classroom observations in 12 schools in
MA, CT, ME, NH, CA, GA, VA

50. What obstacles exist for wiki use in
poor schools?
• Differences in resources?
• Differences in school culture?

51. Differences in networked technology Low Poverty Medium Poverty High Poverty
resources among public school districts Districts Districts Districts (>20%)
(<10%) (11-20%)
Provide teachers with their own server space for
posting their own Web pages or class materials
(Elementary 90% 81% 74%
Secondary) 92% 84% 74%
Provide students with electronic storage space 76% 60% 50%
on a server 92% 85% 72%
Provide students with online access to the 82% 69% 66%
library catalogue 92% 82% 72%
Provide students online access to databases (for 71% 58% 53%
library resources) 79% 67% 57%
Employ an individual responsible for education
technology leadership
(Full Time 60% 48% 47%
Part Time 26% 35% 33%
None) 13% 17% 20%
SOURCE: U.S. Department of Education, National Center for Education Statistics, Fast Response Survey System (FRSS),
“Educational Technology in Public School Districts, Fall 2008,” FRSS 93, 2008.

52. Fear, anxiety and worry about student
exposure to the Internet
Barbara: There was - the superintendent said, "Do want
you want but I never want us on the frontpage with some
bad headline" [chuckle]. I thought those are kind of
interesting. That's why he told our technology committee.
Our main goal is to not make the frontpage with the bad
headline attached.
Jim: And as far as I know but I don't want to be the person
to be in the newspaper for... Look at what this kid was
doing on the internet in a school. I don't want that

53. Framework for Intervention:
Cycle of Experiment and Experience
Experiment
Fear - Growth+
Review
Plan
(Experience)
Institutional Capacity+

54. The atmosphere at our school in general is to cautiously open up to the
possibilities. So, we’re not being pushed to use technology. If we find our way
to it, and if we find obstacles that are there and we need things to change to get
access to certain things, the administration generally will make that happen. But
this, I think in terms of the environment, security is more important than
openness in general and sometimes that leads us into a few obstacles….
We haven't gotten to the point where we're making a lot of the stuff public yet.
Is that possible down the road? I think with a little bit of experience, with a
little bit of, we improve the comfort level then, we can start to say, “Let's make
a blog that invites other people throughout the world. Whoever wants to come
visit and check it out and be part of the conversation; let's figure how to do
that.” I don't think I should... I'm not really in that place that right now. I'm not
sure what the response would be if I did. I think we all sort of need to build that
comfort level piece first. But I also see that it could be pretty awesome to move
in that direction.

55. Dear Justin,
I am in the library right now and sat down to do some work on
the Wikispace to get it ready for next year. However, upon
sitting down, I discovered that over the summer this website
has been blocked by the City. I spoke to the librarian about
appealing blocked content, but he says that they are not
unblocking any sites at this time.
Consequently, it does not look like we are going to be doing
this project this year. I will be trying to put together
something else for this unit, but at this point, I don't know
what we will be doing and doubt it will be appropriate for
your project because it will not be on wikispaces.com.
I am very sorry; it was quite a surprise to me.

56. Framework for
Inequities in wiki usage
Wikis created in non-
Intervention:
suggest that Web 2.0
poor schools persist
Promoting a Cycle of
tools may than and
longer exacerbate
Experiment
wikis
nd digital schools
created in poor divide
the 2Experience
Experiment
Fear - Growth+
Non-poor students
21st Century Skills
Review
Poor students Plan
(Experience)
Institutional Capacity+

57. What Next?
1. Develop an instrument to measure Wiki
Quality
2. Correlate wiki quality profiles with teacher
attitudes, practices, and resources
3. Develop computational tools to automate
those analyses at scale
4. ?
5. Profit

58. Acknowledgements
• Hewlett Foundation Open Education
Resources for grant support
• PBworks for data support
• Hunter Gehlbach, Stone Wiske, Laura
Schifter, Anna Savaadra, and other
readers of this paper.
• Benjamin Mako Hill for coming up the river
to offer his thoughts and critique! 58