Current challenges for educational technology research Mayes described educational technology research as being like the film, 'Groundhog Day', with "cycles of high expectation [...] followed by proportionate disappointment", and "a cyclical failure to learn from the past". Fifteen years on, this experience still rings true. Is this pattern inevitable and inescapable? This paper identified several challenges faced by work in this area. Together, they go some way towards explaining this pattern, and identifying what will need to change if we are to break out of this. These challenges include the strategic difficulty of maintaining research work across cycles of new technology; the methodological challenge of studying things people have forgotten they are using; the epistemological challenge of reconceptualising the relationship between technology, users and effects; the practical challenge of knowing our learners; and the political challenge of securing funding for anything other than instrumental, applied work. ---- Seminar at Oxford education department, 17/11/10. Cited papers listed in the speaker's notes.

1. Current challenges for educational
technology research
Martin Oliver

2. What I’m going to cover…
• Technology as revolution
• Technology as cause
• The myth of learners
• Technology as solution
• Hope…?

3. • Technology as revolution

4. In the film "Groundhog Day", the protagonist is
forced to experience the events of a single day
over and over again. He is free to act in any way
he chooses, but whatever he does the day
always finishes in the same way.
Part of the fascination of this predicament is the
awful familiarity of this experience: so often one
feels caught in a flow of events which will unfold
in an entirely predictable way.

5. People who have been involved over any length of time
with educational technology will recognise this
experience, which seems characterised by a cyclical
failure to learn from the past. We are frequently excited
by the promise of a revolution in education, through the
implementation of technology. We have the technology
today, and tomorrow we confidently expect to see the
widespread effects of its implementation. Yet, curiously,
tomorrow never comes. We can point to several
previous cycles of high expectation about an emerging
technology, followed by proportionate disappointment,
with radio, film, television, teaching machines and
artificial intelligence.

6. Revolutions
• Multimedia (CD roms)
• Hypertexts
• World Wide Web
• Virtual Learning Environments
• Wikis, blogs
• Social networks, virtual worlds, games,
twitter…

7. Revolutions or cycles?
• Proof of concept (needed!)
• Replication
• Replication in disciplinary contexts
…increasingly “me too” studies
very few, “yes, but” or “no, not really”
studies

8. What do I see getting published?
• Lots of (positive) case studies
• A fair amount of studies of organisational
development (inc. academic/staff
development)
• Some studies of design/development
• Some quasi-experimental studies (quantitative but still
case-based)
• A few multi-site comparative studies (qualitative and quantitative)
• Some national surveys
• Some theory, position papers etc

9. So how should we be looking at this?
• Do ‘medical’/“agro-botanical” studies better
(Alsop & Tompsett, 2007)
1 Can a new drug be shown to have an effect? Effect
2 Ditto, with an RCT in a selected population that takes
drug properly?
Efficacy
3 Ditto, for a typical population that behaves normally Effectiveness
4 Any previously unobserved outcomes post release? Side effects

10. …or…?
• But…
– Still dose/response assumption
– Assumes aggregation is unproblematic
• Do something different
– Design research
– Thesis/antithesis/synthesis theory building
– Something else…?

11. None of it makes any difference anyhow…
• http://www.nosignificantdifference.org/
• The phrase refers to the innumerable quasi experimental
“media comparison” studies […] These studies have
shown, in other words, that technology-based courses,
when compared to those taught in the classroom, do not
result in a statistically significant difference in student
performance or educational efficiency. From print-based
correspondence to courses taught via radio, television,
and the Web, the use of new media in each case was
not found to result in a statistically significant
improvement in educational efficiency.

12. • Technology as cause

13. The problem with dose/response models
• “black box” trials – “does it work?”
– Attribution is the object of study
– Causation is problematic
– Explanation is unlikely
• What happens if you open the “black
box”…?

14. Why the medical model never worked anyhow
…analyses [that] comprise an attach on the
philosophical principles behind the evidence-based
Medicine movement. [… It] undermines the assumption
that results from random controlled trials can be applied
across varying context without considering the social and
structural specifics of the context in which medicine is
practiced. […] ‘technological guidelines’ can be
problematic if they are posited to be universal while the
practice they are meant to guide is very place and
culture specific.
(Johnson & Berner, 2010: 76-7)

15. Perhaps we should be studying practices?
• Social studies of science and medicine
– It’s not just “stuff” acting on “stuff” unproblematically
– Scientific knowledge is socially constructed
– But saying it’s all just talk misses out the hands,
bodies and so on of medical practice
…so how can we bind them back together?
• Mol’s ‘Praxiology’ – the study of practices
– Talk, action and their ongoing negotiation

16. Why are there (apparently) no successes?
• Activity theory
– Activity
– Action
– Operation
• Routine, breakdown and repair
• Tacit knowledge
• Anything that works, we become unable to
talk about

17. How does something “work”?
• Studying teachers (HE) starting to use a
virtual learning environment
I’m looking for some kind of contribution, any
contribution, I look for basically and if I don’t get that then
I know there’s probably something wrong. It’s when
people are chipping in their bits and then all of a sudden
it goes quiet. That’s the danger sign. You do pick up on
odd stuff like that – its just transferring what you normally
do in normal situations to a virtual environment.

18. • Activity (Strategy) – no real change
…to responsibilities, values, role
• Action (Tactics) – some differences
…changes to tools used to ‘look’ for participation,
changes to pace and time of ‘looking’
• Operation – completely different
…scan the room, listen for pauses…
…click to generate list of contributors, look for 0’s…

19. What this reveals is how the move to
teaching online renders the role of the
teacher both the same and different
simultaneously. The purpose and strategic
direction may remain unchanged, but the
methods of achieving this alter in
significant ways.
(Price & Oliver, 2007: 24)

20. • The myth of learners

21. Who’s practicing what, exactly?
• Prensky, Tapscott, Oblinger & Oblinger
– We know our learners, and they’re not us
(even down to how their brains work)
“rather than being empirically and theoretically
informed, the debate can be likened to an
academic form of a 'moral panic’”
(Bennett et al)

22. We don’t know our learners…
…but at least we’re trying.
• UK-based studies of learner experiences
of e-learning
– JISC learner experience projects
– ELESIG.ning.com
– Institutional studies: what are our learners like?

23. • National/international studies
– In the UK, it’s not so simple, although there are trends
(Eynon, 2009; Jones et al, 2010)
– In South Africa, there’s sacrifice to get there and
there’s important pockets of non-engagement
(Czerniewicz et al, 2009)

24. …and what are they doing?
• Not what the headlines suggest…
– Selwyn’s “state of the actual”
– Creativity isn’t the norm (Luckin et al 2009);
only a few children do ‘interesting’ things like
media production
– Not using social networks in a collective
wisdom, ‘connectivist’ sense; instead, gossip,
banter and ‘identity work’ (Selwyn, 2009)
…or not using it at all? Average age of social
networker is 35-44

25. Open and flexible learning?
• Well intentioned, but flexibility can cause
as well as solve problems
(Holley & Oliver, 2010)
• Learners struggling with new barrier
• Unfamiliar expectations
• New cultural norms to engage with

26. • Charles – ‘ideal’ student – colonisation of home
space for study, with notes “as far as you can
see”; no mention of family, constraints;
purchases resources “even when they’re not
essential”
• Kwame – non-engaged – struggled, but tried to
avoid bothering those “in authority” (merely
another resources to Charles); struggled to use
shared access machines; knew answers but not
how to communicate them; eventually helped
when he finds a friend

27. Those with social advantage find it easier to
take advantage of new opportunities;
advantage can be perpetuated, not eroded,
by introducing new forms of learning and
teaching

28. • Refocusing on practice, not technology:
People aren’t using it because it’s
technology; they’re using it because they
see it as useful in the context of something
they’ve chosen to do

29. • Technology as solution

30. Back to determinism…
• Given all this complexity, why have we
spent so much on this?
• The appealing myth of technology as a
solution
• A policy desire?
– More technology means more learning
– I want to improve education, but it’s tough
– But I can buy technology…
– So I can buy improvements to education

31. Not just a caricature…
• New labour policy frames education as an
economic endeavour
(not necessarily cultural or political)
• Technology is presented as being able to make
systems more efficient
• Policy drives…
– Purchase (but now saturation)
– Some innovation (but the exception not the norm)
– Scapegoating of teachers for non-use (it’s not that
just buying tech didn’t work, honest…)
(Saima Rana)

32. A massive influence on research too
• Funding from JISC, The Higher Education
Academy
– Proof of concept, implementation projects,
quality enhancement (and learner experience)
(This explains a lot about the technological
emphasis/patterns of publication earlier…)
– Even medical model fails: roll-out isn’t a
(fundable) research issue

33. “To use the words of educational technologist Rob Koper
[…] this research tends not to be “theory-oriented,” but
rather “technology-oriented” in character. E-learning
research, Koper (2007) explains, is not focused on
“predicting or understanding events [in] the world as it
exists” (p. 356); it instead seeks to “change the world as it
exists” (p. 356; emphasis added). E-learning or technology-
oriented research, in other words, attempts “to develop new
technological knowledge, methods, and artifacts” for
practical ends or purposes (p. 356). It is this applied,
practical, and technological research that Koper (2007) says
is ideally suited to e-learning.”

34. • Knowledge-constituative interests
– Technical (prediction and control)
– Practical (mutual and self understanding)
– Emancipatory
(New ALT SIG: Politics, activism and
critical theory)

35. To illustrate some of this…
• Open Educational Resources
OER are teaching, learning, and research resources
that reside in the public domain or have been
released under an intellectual property license that
permits their free use or re-purposing by others. Open
educational resources include full courses, course
materials, modules, textbooks, streaming videos,
tests, software, and any other tools, materials, or
techniques used to support access to knowledge.

36. How does this fit with Higher Education?
• Echoing moves such as MIT Open
Courseware initiative
• In the UK, part of a growing tradition of
research and policy
– Projects, e.g. OU Learning Design Initiative
– Policy about how content should be created
and shared, and how learning should be
organised

37. Dearing report (1997)
• Analysis of technology-related excerpts
– Students portrayed as passive, apart from
when choosing a course (described in terms
of costs and outcomes), then ‘developed’
– Lecturers not talked about as teachers;
expectation of giving up teaching to focus on
materials development
– Technology described in terms of access to
information

38. The new interactive media, offering adaptive feedback
and student control have the potential to support
independent study, but only if fully developed, tested and
maintained. […] Many staff would seek to spend some of
their time on development of learning materials, because
these will enshrine the core of their teaching. […] IT
methods must achieve their promise of greater efficiency
both by improving the quality of student learning, and by
amortising the cost of development over large student
numbers.
(Dearing report, Appendix 2)

39. What constitutes the curriculum here?
• Curriculum as syllabus (a thing to be
bought)
• Curriculum as materials (to be consumed)
• Some pedagogy (Resource-based
learning)
• Gives no real sense of the experience of
learning

40. The e-University
As the learner progresses through the courseware, there
is the opportunity to ask questions by selecting the
associated ‘chat’ channel in the toolbar. In response, a
chat window opens and the learner is greeted and
invited to describe the assistance sought, in text form.
The person who answers the questions is part of a call
centre and is specifically trained to answer questions
about the courseware. […] If the mentor is unable to
answer a question, it is referred to a tutor with superior
subject expertise, who returns a full answer to the
learner by e-mail within a set period.

41. Not-so-intelligent tutoring?
• Intelligent tutoring systems
– Selecting and sequencing of materials for
each learner
– Built on a (sophisticated) student model
– Big in the 70s/80s, largely abandoned in the
90s as not really worth the effort
(Est. half as good, at best, as tutor support)
• Now, available materials ‘personalised’
using learning styles…

42. Cook recommendation to Denham committee
…a new approach to virtual education based on a
corpus of open learning content: the UK must have a
core of open access learning resources organised in a
coherent way to support on-line and blended learning by
all higher education institutions and to make it more
widely available in non-HE environments. This needs to
be supported by national centres of excellence to
provide quality control, essential updating, skills training,
and research and development in educational
technology, e-pedagogy and educational psychology. All
HEIs should be encouraged and helped to exploit virtual
education technologies as appropriate to their student’s
requirements and their strategies.

43. Building an effective and competitive on-line learning
capacity at both undergraduate and postgraduate level
will help meet the changing needs of students and
stimulate growth in both higher education and the skills
sector. A coherent collection of learning resources can
also be exploited to save staff time. Failure to do so will
reduce the UK’s ability to exploit e-learning; an aspect of
learning and teaching where the UK should aspire to,
and gain, a world leading position.

44. • The UK must have a core of open access learning resources
– Why? If they’re open, why not use US ones, etc?
• This needs to be supported by national centres of excellence
– Not quite the monolithic e-University, but still about centralisation,
control, and (arguably) preservation of advantage
• research and development in educational technology, e-pedagogy
and educational psychology
– But not education, ethics, cultural studies… all instrumental
• help meet the changing needs of students
– Do we really know what these are?
• Failure to do so will reduce the UK’s ability to exploit e-learning
– It’s all about the money

45. What have we learnt?
• Studies about production and tagging of
resources, design of repositories
– Relatively little about successful re-use
• Better ways to reify practice
– Work of formalisation (e.g. LDSE) and pedagogic
patterns
– Relatively little about how teachers have made use of
these
• Processes: of excellent value to researchers, but
perhaps of less impact for teachers

46. • Like Cuban’s studies of technology in schools:
– Belief in the power of technology
– A moral imperative to change
– An economic argument
– Scapegoating: staff have ‘failed’ to do this
• Systematic separation of course production from
academic role
– A power grab in the name of efficiency
– Can be traced back to instructional design and “so
called subject matter experts (SMEs)”

47. What could we learn?
• Millen’s studies of course reading lists
– The politics of curricula
• Small-scale work about the co-ordination
of course texts by ‘exemplary’ VLE users
– Design as performance
– Agility and adaptivity vs. transparency and
encoding; practice, spaces and reification
• Where are the studies of how teachers
work with textbooks?!

48. What could we do?
• Cape Town Open Education Declaration
This […] combines the established tradition of sharing
good ideas with fellow educators and the
collaborative, interactive culture of the Internet. It is
built on the belief that everyone should have the
freedom to use, customize, improve and redistribute
educational resources without constraint. [… P]art of
a worldwide effort to make education both more
accessible and more effective.

49. These resources include openly licensed course
materials, lesson plans, textbooks, games, software and
other materials that support teaching and learning. They
contribute to making education more accessible,
especially where money for learning materials is scarce.
They also nourish the kind of participatory culture of
learning, creating, sharing and cooperation that rapidly
changing knowledge societies need.
However, open education is not limited to just open
educational resources. It also draws upon open
technologies that facilitate collaborative, flexible learning
and the open sharing of teaching practices

50. Hype, hope and disappointment
• Revolutions as repetitive cycles
• Difficulty of shifting naïve accounts of the
influence of technology
• Little (funded) interest in critical,
emancipatory work
• Where now?

51. No funding - is there a bright side?
• UK cutbacks marginalising technology in
education
– As a very ‘new labour’ thing, is it realistic to expect it
to be supported?
• Less research in the area… but will the
research that does happen be better as a
consequence?
– Tapping into established concerns
– Working through other disciplines (any discipline is
better than none!)

52. • Consolidation?
– Less innovation, more valuing (and understanding) of
established practice?
– More scope to synthesise research and genuinely
address gaps, not just follow fashions?
• Less instrumentalism?
– If it’s no longer ‘the solution’, perhaps we can think
differently about it?
– Growing interest in critical engagement – can this be
sneaked in…?

53. Fin.