A presentation on various ways one might try to evaluate the effectiveness of cMOOCs, and some questions and concerns about each one, ending with a question: how best should we do this?
1. Difficulties evaluating
cMOOCs: Negotiating
Autonomy and Participation
#DiffCMOOC
Christina Hendricks
University of British Columbia, Vancouver
Open Education Conference, November 2013
Presentation licensed CC-BY
2. Connectivist
MOOCs
Network: Facilitating
connections between
people and information,
ideas (not transmitting
knowledge from central
source) (Siemens 2012
http://is.gd/K5JfXK )
Distributed: Takes place
in multiple spaces (blogs,
wikis, tweets, discussion
boards, webinars, etc.):
“A MOOC is a web, not a
website” (Downes 2013a
http://www.downes.ca/presentation/327
)
#OOE13 Open Online Experience 2013-2014
http://www.ooe13.org
3. Connectivist
MOOCs when & how to participate;
Autonomy: Participants decide
create own learning goals, choose own paths through course
(McAuley et al. 2010 http://is.gd/6j1X1k; Downes 2009 http://is.gd/AYc84B)
Open: free access
available to anyone
with reliable internet
connection;
curriculum open to
alterations by
participants
(Downes, 2013b
http://is.gd/Downes2013 )
From a video on MOOcs by Dave Cormier & Neal Gillis (licensed CC-BY)
http://is.gd/cQwOSP
4. Evaluating cMOOC
effectiveness
Do they achieve goals?
Which goals?
• Of designers
• Of participants
• Connectivism: making connections w/people &
information
• What sort of entities cMOOCs are & whether fulfill
purposes (Downes)
5. Goals of cMOOC
designers
Participant autonomy:
•What happens in course depends on
what participants do: “learners are
expected to actively contribute to the
formation of the curriculum through
conversations, discussions, and
interactions” (Cormier & Siemens, 2010
http://is.gd/nqTED )
Hub & Spoke, flickr photo by
Antony_Mayfield, licensed CC-BY
•course may be successful (or fail) in
ways designers never envisioned
6. Goals of participants
E.g., Lane, 2013 http://is.gd/W0360s
•participants may have goals that don’t fit course; a
problem if not fulfilled?
• may not have any goals; just want to
see what happens
• course may have other benefits not
captured in participants’ goals; may
miss this if focus on their goals
• benefits may take a long time to
realize
Huma Bird tweet analysis,
#whyopen http://is.gd/4h5CFq
7. Goals of participants
What one might do:
•Ask participants at the end what they got out of the
course, with or without reference to their original goals
•Return to them six months or more later to ask again-perhaps see longer-term effects
•Consider how to support learners in being self-directed,
working to achieve own goals in a cMOOC (e.g., Kop, Fournier
and Mak, 2011 http://is.gd/KopEtAl2011 )
8. Connectivism:
connections among
people & info
• “Knowledge is defined as a particular pattern of
relationships and learning is defined as the creation of new
connections and patterns as well as the ability to maneuver
around existing networks/patterns." (Siemens 2008
http://www.connectivism.ca/?p=116 )
• "At its heart, connectivism is the thesis that knowledge is
distributed across a network of connections, and therefore
that learning consists of the ability to construct and
traverse those networks." (Downes, 2007 http://is.gd/Downes2007)
9. Connectivism:
connections among
people & info
Participation rates:
•surveys of participants: Milligan, Littlejohn and Margaryan,
2013 http://is.gd/MilliganEtAl2013
•log data from P2PU platform: Ahn, Weng and Butler, 2013
http://is.gd/AhnEtAl2013
•mixed methods:
• Waite, Mackness, Roberts and Lovegrove, 2013
http://is.gd/WaiteEtAl2013
•
Kop, 2011 http://is.gd/Kop2011
10. Connectivism:
connections among
people & info
Negotiating autonomy
& participation
•Tension: need at least
some active
participation, but
participants must have
autonomy
•Lurkers valued? Just b/c
may become active
participants?
#ds106zone May 25-June 6, 2013
http://is.gd/o27mvc
11. Purposes of cMOOCs
themselves
Downes 2013b http://is.gd/Downes2013
•look at what sorts of entities cMOOCs are, what purposes
they serve, whether designed well for those purposes (rather
than how they’re used)
•To evaluate a cMOOC, consider: “what a successful
MOOC ought to produce as output, without reference to
existing (and frankly, very preliminary and very variable)
usage.” (Ibid.)
• output: “emergent knowledge”
12. Emergent knowledge:
See something or
say something: Jakarta, Flickr photo shared by
Eric Fischer, licensed CC-BY
Blue dots tweets; red dots Flickr,
white dots both
In a successful cMOOC, “the
structure of the interactions
produces new knowledge, that is,
knowledge that was not present
in any of the individual
communications, but is produced
as a result of the totality of the
communications, in such a way
that participants can through
participation and immersion in
this environment develop in their
selves new (and typically
unexpected) knowledge relevant
to the domain.” (Downes, 2013b;
emphasis added)
13. Networks that tend to
produce emergent
knowledge
1.Autonomy
2.Diversity
3.Openness
4.Interactivity/Connectedness
(Downes, 2013b)
http://is.gd/Downes2013
Anek Rang, Ek Sang, Flickr photo
shared by Sanjay, licensed CC-BY
14. How evaluate
cMOOCs acc to
these criteria?
• Don’t measure each aspect of a cMOOC against these
as if a checklist; rather, consider cMOOCs a
“language” and a course as an expression in it
• These criteria should be considered “an aid, used to
assist a person who is already fluent in MOOC
design (or at least in the domain or discipline being
studied) [to] recognize the quality (or lack of quality)
of a MOOC” (Downes, 2013b).
15. Questions & concerns
about this approach
• How can we determine if emergent knowledge has been
produced? Where would we look? Whom would we ask?
• Seemingly exclusive focus on
design and purpose of cMOOCs-doesn’t consider the experiences
of participants
E.g., participant experiences in a
cMOOC: Mackness, Mak and
Williams, 2010
Crowd, Flickr photo by James Cridland, lic http://is.gd/MacknessEtAl2010
ensed CC-BY (altered)
16. Footprints of
emergence
•
Williams, Karousou and Mackness, 2011
http://is.gd/WilliamsEtAl2011
•
Emergent and prescriptive learning--need balance
Williams, Mackness and Gumtau, 2012
http://is.gd/WilliamsEtAl2012
-
Draw “footprints” of courses to map degrees of
prescriptive and emergent learning
17. Footprint for CCK08
published in Williams, Mackness & Gumtau
2012 http://is.gd/WilliamsEtAl2012
(licensed CC-BY)
Centre: prescriptive learning
Light area: apex of emergent
learning
Periphery: “edge of chaos”
Map points based on 24
factors, in four clusters
See wiki for factors, how to draw
footprints, and more:
http://footprints-of-emergence.wikispac
/
18. Full circle
• Footprints are not meant to provide evaluations of
courses by themselves
• Instead, provides way to evaluate if course fits purposes
(Footprints of emergence wiki: http://footprints-of-emergence.wikispaces.com/ )
• After have footprint, ask: “Is this appropriate, or fit, for
the purpose and context of the course and for you,
and/or the particular learners?”
Back to the beginning...
autonomy: do people make their own decisions about goals and objectives? Do they choose their own software, their own learning outcomes?
-- “members of the network … employ their own goals and objectives, judgments and assessment of success in the process of interaction with others” (Downes, 2013c)
diversitydo people speak different languages, come from different cultures, have different point of view, make different software selections, access different resources? If everybody does the same thing, then nothing new is generated by their interacting with each other; but if they are diverse, then their participation in the network produces new knowledge”
oppenness: “does communication flow freely within and without the network, is there ease of joining (and leaving) the network? In a community, this means, are people able to communicate with each other, are they easily able to join the community, are they easily able to participate in community activities?
Interactivity: “is the knowledge produced in the network produced as a result of the connectedness, as opposed to merely being propagated by the connectedness? If a signal is merely sent from one person to the next to the next, no new knowledge is generated. Rather, in a community that exhibits connectivist dynamics, knowledge is not merely distributed form one person to another, but is rather emergent from the communicative behaviour of the whole. The knowledge produced by the community is unique, it was possessed by no one person prior to the formation or interaction in the community.”
-- “the act of learning a discipline--a trade, for example, or a science, or a skill--is more like the learning of a language than it is like learning a set of facts. ... the bulk of expertise in a language ... [is] in fluency and recognition, cumulating [sic] in the (almost) intuitive understanding .... This sort of fluency is acquired by immersion in a language-speaking community (of which a MOOC is a characteristic example)” (Downes, 2013c)
emergent; “learning which arises out of the interaction between a number of people and resources, in which the learners organise and determine both the process and to some extent the learning destinations, both of which are unpredictable.
— knowledge “created and distributed largely by the learners themselves” (Ibid., 43)
"Emergent learning is likely to occur when many self-organising agents interact frequently and openly, with considerable degrees of freedom, but within specific constraints; no individual can see the whole picture; agents and system co-evolve." (Ibid., 45)
presccriptive: hierarchical, transferred from central knowledge source to those who are going to learn, knowledge is pre-determined for learners
2. need balance of openness and constraint: if too open can become chaotic, people get lost and drop out, or people can end up in echo chambers b/c dissenting voices may not be heard;
— need to have constraints to ensure inclusive, inviting environment for ppl to share, discuss, so learn from one another, try to ensure dissenting voices don’t get drowned out