Keynote at “Celebrating 15 years of Promoting Excellence in Teaching and Learning”, 25-27 February 2018, American University of Cairo, Egypt

1. Paul Prinsloo
University of South Africa (Unisa)
14prinsp
How do we know they learn?
The changing scope and role of
evidence in student learning
Image credit: https://pixabay.com/en/building-cctv-door-female-ladies-1839464/
Keynote at “Celebrating 15 years of Promoting Excellence in
Teaching and Learning”, 25-27 February 2018,
American University of Cairo

2. Imagecredit:https://pixabay.com/en/binary-code-man-display-dummy-face-1327512/
I do not own the copyright of any of the images in this
presentation. I therefore acknowledge the original
copyright and licensing regime of every image used.
This presentation (excluding the images) is licensed
under a Creative Commons Attribution 4.0 International
License.

3. Imagecredit:https://pixabay.com/en/binary-code-man-display-dummy-face-1327512/
Overview of the presentation
• Thinking about laptops in the classroom and
dinosaurs
• Dreaming of losing control
• Considering why we need/want/have to know that
students are learning
• A case study of broken data, broken lenses and
broken hopes
• (In)conclusions

4. Fall 2017
Source credit: http://educationnext.org/should-professors-ban-laptops-classroom-computer-use-affects-
student-learning-study/

5. Source credit: http://educationnext.org/should-professors-ban-laptops-classroom-computer-use-affects-
student-learning-study/
“we do not claim that all computer use in the
classroom is harmful. Exercises where computers
or tablets are deliberately used may, in fact,
improve student performance. Rather, our
results relate to classes where using computers
or tablets for note-taking is optional.”
Fall 2017

6. Source credit: https://www.nytimes.com/2017/11/22/business/laptops-not-during-lecture-or-meeting.html
22 November 2017
“Measuring the
effect of
laptops on
learning is
tough. One
problem is that
students don’t
all use laptops
the same way.”

7. 22 November 2017
“But a growing body of evidence shows that over all,
college students learn less when they use
computers or tablets during lectures. They also tend
to earn worse grades. The research is unequivocal:
Laptops distract from learning, both for users and
for those around them. It’s not much of a leap to
expect that electronics also undermine learning in
high school classrooms or that they hurt
productivity in meetings in all kinds of workplaces.”

8. Haven’t we been here before?
Image credit: https://pixabay.com/en/shock-woman-portrait-hand-2757395/

9. 11 September 2016
Source credit: https://www.chronicle.com/article/No-Banning-Laptops-Is-Not-the/237752

10. Source credit: http://activehistory.ca/2014/07/a-brief-history-of-the-laptop-ban/
From 2011 – Duke University – “If classroom time is
primarily characterized by lectures, then laptops are
probably a distraction, and if students bring them to
class for note taking, it is quite likely that some will
use them for activities unrelated to the class.”
31 July, 2014

11. Image credit: https://pixabay.com/en/dinosaur-mirror-wing-mirror-behind-1564323/
The object in the rearview mirror may be
closer than it is… – and it is not technology…

12. What may be haunting us, isn’t that they are not
learning, but that we are not solely in control of
their learning (as if we ever were…)
Image credit: http://geopoliticsrst.blogspot.com.eg/2014/12/isis-saudi-arabia-puppet-master-of-isis.html

13. Image credit: https://pixabay.com/en/women-lay-monochrome-soledad-one-3118387/
Losing control is not the only reason why many of
us wonder: “But are they learning?”

14. Image credit: https://pixabay.com/en/sculpture-bronze-the-listening-2275202/
So, why do we need/want/have to know that
they are learning?

15. Why do we need to know that they learn?
Because…
1. We can know
2. Metrics make the world go round
3. We know or the unbearable lightness of arrogance
4. We have the tools
5. Our students are broken and need fixing
6. We want to understand, listen and act
7. We ant to understand where/how we can help
them to do better

16. Image credit: https://pixabay.com/en/turn-on-turn-off-industry-energy-2923046/
Reason 1: Because we can know

17. Source credit: https://www.amazon.com/Owned-Property-Privacy-Digital-
Serfdom/dp/1316612201/ref=sr_1_4?ie=UTF8&qid=1519453392&sr=8-4&keywords=owned
As learning moves increasingly online, we
(Silicon Valley, governments, data
brokers, commercial entities, and higher
education) have access to huge amounts
of intersecting, and mutually constitutive
data sets. And we own their data. And
we are owned.
The learning management systems we
use have become a treasure trove of
student data that we can process, scrape,
clean, dissect, mine, massage to find
patterns, and at times, make to fit our
assumptions about whether they are
learning or not

18. We have access to ever increasing volumes, velocity and
variety of student digital data, that allows us to expand not
only on the traditional scope of institutional research with
regard to student data, but also to infer relations
unthinkable ten years ago. We may therefore be tempted to
rush to look for patterns without considering our own
assumptions and epistemologies
There is a real danger to mistake the noise for a signal, and
not realise that the noise pollutes our data with false alarms
and “setting back our ability to understand how the system
really works” (Silver, 2012, p. 162)

19. Reason 2:
Because
metrics make
the world go
round

20. Image credit: https://www.amazon.com/Metric-Power-David-
Beer/dp/1349717681/ref=sr_1_1?s=books&ie=UTF8&qid=1519507130&sr=1-1&keywords=metric+power
“We are created and recreated by
metrics; we live through them, with
them, and within them. Metrics
facilitate the making and the remaking
of judgements about us, the
judgements we make of ourselves and
the consequences of those
judgements as they are felt and
experienced in our lives. We play with
metrics and we are more often played
by them” (Beer, 2016, p. 3)

21. Image credit: https://pixabay.com/en/tape-measure-wrap-up-hands-finger-2406650/
“Measures define what is true and then are
used to verify the truth” (Beer, 2016, p. 28)

22. Image credit: https://pixabay.com/en/justitia-goddess-goddess-of-justice-2597016/
We can only know when
we measure and when
we know we have the
power to to claim
understanding, to define
what is ‘true’, and based
on this ’truth’, we assign
positions, class, potential,
and futures that often
lock individuals into
fulfilling our ‘truths’
about them

23. Image credit: https://memeguy.com/photo/12358/theres-always-that-one-guy-at-work
By measuring and allocating value, we can find those
who do not fit in our normal distribution curves, and
isolate them and define them as abnormal, not normal

24. “Evidence-based education seems to favour a
technocratic [and quantitative] model in which it is
assumed that the only relevant research questions
are about the effectiveness of educational means
and techniques, forgetting, among other things,
that what counts as “effective” crucially depends
on judgments about what is educationally
desirable” (Biesta, 2007, p. 5)

25. Source credit: http://onlinelibrary.wiley.com/doi/10.1111/j.1741-5446.2006.00241.x/full

26. Source credit: https://link.springer.com/article/10.1007/s11217-010-9191-x

27. Image credit: https://pixabay.com/en/pot-sticks-fire-smoke-ritual-2567719/
Our institutional rituals of verification sanction those of
us who measure frequently, those of us who are
measured frequently, and for all of us who submit our
reports to those who need (our) data to exist

28. Image credit: https://pixabay.com/en/sculpture-bronze-the-listening-2209152/
Reason 3: Because we know…
… everything

29. Image credit: https://www.pexels.com/photo/big-data-business-cloud-communication-236906/
Where the institutional LMS resembles not
only a walled garden…
…but increasingly Foucault’s Panopticon where our
tracking systems and recommender engines watch
over their every move

30. Page credit: http://www.npr.org/sections/ed/2017/01/11/506361845/the-higher-ed-learning-revolution-tracking-each-
students-every-move
2017

31. • We feed our data fetish, believing that more data are always
better data
• We believe that big(ger) data gives us a total picture where
n=all and that there is nothing we cannot know
• We believe that data are neutral and objective and not as
subjective as human decisions based on experience,
intuition and wisdom
• We create data proxies for the details of our students’ lives
for which we do not have data, or that we cannot quantify
• We think finding patterns is enough and we never care to
ask ‘why’?
And because we think we (can) know
everything…

32. Apophenia – “seeing patterns
where none actually exist,
simply because enormous
quantities of data can offer
connections that radiate in all
directions”
(boyd & Crawford, 2012, p. 668)
Our epistemological arrogance leads us to
see Jesus in a piece of burned toast
Seeing Jesus in toast: Irreverent ideas on
some of the claims pertaining to
learning analytics (Prinsloo, 2016) –
https://opendistanceteachingandlearning.wo
rdpress.com/2015/12/07/seeing-jesus-in-
toast-irreverent-ideas-on-some-of-the-claims-
pertaining-to-learning-analytics/
@Jesus_H_Toast

33. Citation: Pink, S., Ruckenstein, M., Willim, R., & Duque, M. (2018). Broken data: Conceptualising data in an
emerging world. Big Data & Society, 5(1), 2053951717753228.

34. Image credit: https://pixabay.com/en/camera-spy-pigeon-surveillance-712122/
Reason 4: Because we have the
tools…

35. Three sources of data
Directed
A digital form of
surveillance
wherein the
“gaze of the
technology is
focused on a
person or place
by a human
operator”
Automated
Generated as “an
inherent,
automatic function
of the device or
system and
include traces …”
Volunteered
“gifted by users
and include
interactions
across social
media and the
crowdsourcing of
data wherein
users generate
data” (emphasis
added)
Kitchen, R. (2013). Big data and human geography: opportunities, challenges and risks. Dialogues in Human
Geography, 3, 262-267. SOI: 10.1177/2043820613513388

36. Source credit: http://timoelliott.com/blog/2013/02/gartnerbi-emea-2013-part-1-analytics-moves-to-the-core.html
Getting to prescribe on the
basis of ‘students like you’…

37. Image credit: https://pixabay.com/en/people-man-guy-camera-alone-photo-2587110/
Reason 5: Because our students are broken and
need fixing

38. Source credit: https://www.edsurge.com/amp/news/2017-11-13-what-can-higher-ed-learn-from-precision-medicine
“Focused hospitals are built around a very specific
value-adding process activity: They take incomplete
or broken parts and then transform them into more
complete outputs of higher value while charging a fee
for the outcome.”
Higher education should be like hospitals
using precision medicine…

39. Image credit: https://commons.wikimedia.org/wiki/File:I_Dont_Know_What_to_Call_This..._(8455760157).jpg

40. Image credit: https://www.flickr.com/photos/highersights/6231641551
Because we want
to…
Reason 6

41. Image credit: https://pixabay.com/en/friends-trust-friendship-together-1015312/
We want to…
• Know what we (not our
students) can do
different, better or stop
doing
• Know how we can and
when we should step
back, step in, not do
anything, intervene,
nudge and care

42. • We know that we cannot cause learning to happen
and only create spaces and occasions where learning
can happen
• We know that there is a lot of learning that we plan
for that does not happen. Some of the learning we
plan for may happen (often despite our failures and
missed opportunities
• And we also know that there is also a lot of learning
that we did not plan for, that actually happens
Image credit: https://pixabay.com/en/guy-man-male-people-child-boy-2609607/

43. Image credit: https://pixabay.com/en/potter-hands-sound-tonkunst-2106654/
Reason 7: We want to understand where/how
we can help them to do better

44. Image credit: https://pixabay.com/en/wonder-woman-superman-superhero-552109/
Beware: They don’t need
superwoman or superman

45. Imagecredit:https://pixabay.com/en/binary-code-man-display-dummy-face-1327512/
Image credit: https://pixabay.com/en/person-male-man-portrait-shadow-828630/
“If you have come to help us, you can go
home. If you have come to accompany us,
please come. We can talk”
Glesne, C. (2016). Research as solidarity. In T. Kukutai and J. Taylor. (Eds), Indigenous data
sovereignty. Toward an agenda (pp. 169-178). Canberra, Australia: Australian National University
Press. Retrieved from https://press.anu.edu.au/publications/series/centre-aboriginal-economic-
policy-research-caepr/indigenous-data-sovereignty

46. Image credit: https://pixabay.com/en/light-out-of-focus-focus-lamps-1068944/
What are the implications when we use their data
points to describe, diagnose, predict and prescribe
their learning journeys without ever asking them
what these data points mean to them, and what
data would matter to them to allow them to make
more informed decisions, to complete their journeys
and not ours?

47. Image credit: https://pixabay.com/en/camera-surveillance-camera-1778861/
Responsible and ethical learning analytics is found in
the nexus between their stories and ours. We cannot
afford to ignore the fact that it is their data, their
aspirations, their learning journeys and that our data
collection, analysis and use may not tell the whole
story.

48. Image credit: https://pixabay.com/en/listen-to-wall-listen-3014329/
Tell me again, why you want to measure
their learning?

49. Image credit: https://pixabay.com/en/camera-facade-wall-spy-film-1570429/
But do they learn? A
case study of broken
data, broken lenses
and broken hope

50. Three weeks into a 12-week, fully
online course: Are they learning
(according to our structure, our
criteria and what we can and want
to measure)?

51. Exhibit A

52. Exhibit B

53. Image credit: https://pixabay.com/en/person-listening-headphones-eyes-2157916/
What does this mean?

54. Exhibit C

56. Exhibit D

58. Student Z
At the end of a 12-
week course: Have
they learned
(according to our
structure, our
criteria and what
we can and want to
measure)?
Exhibit E

59. Image credit: https://pixabay.com/en/buddha-statue-stone-statues-speak-546458/
If this data tell a story, whose story is it?
Is this the whole story? Who is listening?
What did me miss?

60. Prinsloo, P. (2016, November 7 ). Failing our students: not noticing the traces they leave behind. [Web log post]. Retrieved from
https://opendistanceteachingandlearning.wordpress.com/2016/11/07/failing-our-students-not-noticing-the-traces-they-leave-
behind/

61. I thought I cared.
But I did not notice her/him falling behind. I
had access to data abut her/his engagement.
And somehow did not notice
Prinsloo, P. (2016, November 7 ). Failing our students: not noticing the traces they leave behind. [Web log post]. Retrieved from
https://opendistanceteachingandlearning.wordpress.com/2016/11/07/failing-our-students-not-noticing-the-traces-they-leave-
behind/

62. Imagecredit:https://pixabay.com/en/statue-camouflage-
hidden-discreet-2393168/
“… when a world is reduced to numbers, a measure, to
what is calculable and laid before us; when humans are
summed, aggregated and accounted for; then much
remains forgotten, unsaid, concealed”
(Elden, 2006, in Beer, 2016, pp. 59-60)

63. Image credit: https://pixabay.com/en/sculpture-bronze-the-listening-2275202/
Please tell me again, why do you want to know
that they are learning? And how will you know?

64. Paul Prinsloo (Prof)
Research Professor in Open Distance Learning (ODL)
College of Economic and Management Sciences, SP3-15, P O
Box 392, Unisa, 0003, Republic of South Africa
T: +27 (0) 12 433 4719 (office)
T: +27 (0) 82 3954 113 (mobile)
prinsp@unisa.ac.za
Skype: paul.prinsloo59
Personal blog:
http://opendistanceteachingandlearning.wordpress.com
Twitter profile: @14prinsp
Thank you