This is a review of different approaches to eLearning research. The presentation makes the point that we need to conduct research that makes a difference in the classroom and it favors action research in this respect.

1. 1
Dr. Iain Doherty
Director, eLearning Pedagogical Support Unit
Centre for the Enhancement of Teaching and Learning
The University of Hong Kong
December 6, 2013
The Educational Gold Standard
What Can We Hope To Achieve with eLearning Research?

2. 2
GOLD STANDARD
THE GOLD STANDARD IN EDUCATIONAL RESEARCH

3. 3
WHY QUESTION?
Good question . . .
HKU is developing three Massive Open Online Courses (MOOCS), one of
which is being offered by the Li Ka Shing Faculty of Medicine.
MOOCS are throwing up a lot of questions and whilst past research offers
some answers there is clearly a need for MOOC specific research.
I sit in lots of meetings discussing research and I ask myself . . .
What kind of research should
we be conducting?

4. 4
GOLD STANDARD
The idea of a gold standard brings to mind notions of clarity of hypotheses,
adequacy and rigor of research methods, robustness of conclusions etc.
These are not the issues that are addressed in this presentation . . .
At least not directly.
The issue for this presentation really concerns the goals of eLearning /
educational research.
But we should note that
quality of research has been
an issue in eLearning . . .

5. 5
SAD TRUTHS
SOME SAD TRUTHS ABOUT
ELEARNING / EDUCATIONAL RESEARCH

6. 6
Reeves (2000) talks about the generally shoddy nature of eLearning
research and cites substantial evidence for poor methodology and lack of
impact, for example:
“After reviewing nearly 500
papers related to
hypermedia and learning,
Dillon and Gabbard (1998)
identified 118 studies that
appeared to meet their
criteria for quantitative
studies examining the
effectiveness of hypermedia
in education . . . only 30
studies published between
1990 and 1996 met the
minimal criteria of scientific
merit for inclusion in the
literature review”.
Hypermedia and Learning
500 papers
Effectiveness of Hypermedia
118 studies
Minimal Criteria - Scientific Merit
30 studies
(Reeves, 2000)
Review showed little impact from
use of hypermedia

7. 7
Even Worse…
Reeves says that
eLearning / educational
research tends to
occupy the bottom left
quadrant:.
Research inspired by considerations of use
No
Yes
Basic Research
Design Research
Yikes
Action Research
Yes
Research inspired by quest
for fundamental understanding
No
So, we need to remain cognizant of the quality of eLearning research.
Assuming quality – for now at least – we can take a straightforward
perspective on the gold standard for educational research.

8. 8
The Easy Answer
There is a straightforward answer to the question
ANSWER
ONE
The gold standard is
research that improves the
teaching and learning
experience
ANSWER
TWO
This has to be true not just
in a local context but in
multiple contexts
The research has to
provide “theoretical”
insights as a basis for
further research?????
ANSWER
THREE
The improvements also
have to be sustained over
time and they have to be
scalable

9. 9
The Easy Answer made Difficult
Theory
– “An idea or set of ideas that is intended to explain
facts or events”
– “the general principles or ideas that relate to a
particular subject”
Keep in mind the question whether
educational research has to contribute to
educational theory.

10. 10
Direction
For Reeves we need to attend both to what we are doing and to
why we are doing it . . . (Reeves, 2000)
 Assuming clarity of goals, appropriate methodology, rigorous
application of method, clarity of findings, clear conclusions
 We need research that is immersed in the classroom / comes out
of the classroom
Teaching is messy (Brydon-Miller et al, 2003) and if we want to
research it we need to get our hands dirty.

11. 11
ACTION RESEARCH
SHOULD WE AWARD THE GOLD STANDARD
TO ACTION RESEARCH?

12. 12
Action Research
“In general, the demand from academic staff is for help with design –
for customisable, re-usable ideas, not fixed, pre-packaged solutions.
However, demand is also expressed in ways which emphasise academics’
strong sense of being time-poor. There is no visible demand for complex
methodologies, approaches which require substantial revision of existing
work practices, or methods which require mastery of complex skills or
specialised language”. (Goodyear, 2005, p.2)
“Action research can change current classroom practice. Certainly it is better
than merely presenting the research results of professionals, as teachers
must become personally involved in changing their classroom behavior, and
few teachers become personally involved in an issue of the Review of
Educational Research”. (Hodgkinson, 1957)

13. 13
Action Research is in the classroom and involves systematic
observation and data collection (quantitative and / or qualitative) that
can be used by the practitioner-researcher to improve what is going
on in the classroom.
1
Identify
a problem
2
Create
a solution
3
Implement
4
Evaluate
5
Revise
accordingly

14. 14
University of Wollongong Template Designs
Action Research
can be shared quickly
and easily through e.g.
a database of action
research designs.
(Laurillard, 2008)
Improvements are
sustainable in the local
context by the individual
teacher and scalable
through offering the
template designs to
other teachers.
Learning Activity Management System (LAMS)

15. 15
Some people claim that action research is not
really research but just a form of evaluation.
However, it can be considered to be research if
results are shared so that others may draw their
own inferences. (Reeves, 2000)
Unlike “science” this is not
controlled experimentation that
takes variables into account
ACTION
RESEARCH
(Hodgkinson, 1957)
Unlike “science” work stops at
the practical solution with no
attempt to generalize
(Hodgkinson, 1957)
It sometimes
theorizes against
knowledge base

16. 16
EPSU Action Research
EPSU has engaged in some action research with teachers:
FLIPPED
CLASSROOM
Faculty of Medicine
ONLINE
CASES
Department of
Pathology
LANGUAGE
LEARNING
GRADEMARK
School of Chinese
Faculty of Social
Sciences,
Centre for Applied
English Studies

17. 17
Remember the Question Mark
Improve the
teaching and
learning
experience
Can be applied in
multiple contexts
2009)
(Chandler & Torbet 2003)
However, educational
action research is not
generally concerned with
drawing substantially on
theory.
Action
Research
It can provide
theoretical insights
as a basis for
further research
(Friedman & Rogers,
There are many varieties
of action research and
some do aim for
theoretical insights.
Can be sustained
over time and
is scalable
It is useful but not the
right approach for
researching the
complexity of MOOCS.

18. 18
DESIGN RESEARCH
SHOULD WE AWARD THE GOLD STANDARD
TO DESIGN RESEARCH?

19. 19
Researchers with . . .
[design research] goals
are focused on the dual
objectives of developing
creative approaches to
solving human teaching,
learning, and performance
problems while at the
same time constructing a
body of design principles
that can guide future
development efforts” .
(Reeves, 2000)
Design Research – like
action research – is carried
out in the classroom and this
is important in terms of
testing designs with respect
to what is really happening.

20. 20
Empirical Versus Design Research
– Design research works off of what Sandoval refers to as design conjectures
(2004) = hypotheses.
– Based on familiarity with a body of knowledge + the actual teaching situation
researchers, teachers, students etc. come together to create a learning design
that is expected to behave in certain ways in the classroom.
– The “conjectures” are tested in the real world to determine if what was
conjectured is in fact the case.

21. 21
Example of Design Research
– Conjecture that computer mediated situated learning
environment will lead to knowledge that students [pre-service
teachers in this case] can apply. (Herrington & Oliver, 2000; Herrington
& Oliver, 1999)
– Students develop mathematics assessment strategy.
– Herrington and Oliver identify 9 elements to a situated learning
environment and they ground these elements in the relevant
research literature.

22. 22
Example of Design Research
– So the first point is that design research utilizes a lot of theory.
Very time intensive to put a research based design into practice.
– In this case the design research approach also required the
creation of a complex learning environment rich with resources.
– Take a look at the situated learning element again, this time with
the resources mapped against the element.

23. 23
Run the design
experiment and
capture data from
observing students
interacting with the
multimedia
environment, from
reports produced by
the students and from
questionnaires.
Carry out data
analysis and report
findings.

24. 24
Design Research
The clear benefit
here is that principles
are being developed
and tested out in the
environment that
matters
Research
is complex and
intensive with
a lot of data
gathering
The actual
situation is
incredibly complex
and messy and
multiple variables
will impact on the
learning design
However, there is a
positive here because
this is “real world” and
unexpected insights
occur!

25. 25
Design Research can
Improve the
teaching and
learning
experience
Can be applied
in multiple
contexts [but not
generalizable to
multiple
contexts]
Provides
theoretical
insights as a
basis for further
research
Can be sustained over time and is
scalable? Requires a huge
commitment of time / resources
and tends not to be scalable
because others do not have the
inclination / time to put results into
practice / further the research.

26. 26
CONCLUSIONS

27. 27
OK, so the gold standard depends on perspective but there is one
key issue – the research has to make a difference in practice.
– Action research suits teachers + instructional designers +
management to some degree.
– Design research suits teaching intensive teachers + instructional
designers + management to some degree.
– Basic / theoretical research suits, well, researchers +
instructional designers who draw on it + management who have
to report research outputs.

28. 28
Q&A

29. 29
References
• Brown, A. L. (1992). Design Experiments:
Theoretical and Methodological Challenges in
Creating Complex Interventions in Classroom
Settings. The Journal of the Learning Sciences,
2(2), 141–148. Retrieved from
http://www.jstor.org/stable/1466837
• Brydon-Miller, M., Greenwood, D., & Maguire, P.
(2003). Why Action Research? Action Research,
1(1), 9–28. doi:10.1177/14767503030011002
• Chandler, D., & Torbert, B. (2003). Transforming
Inquiry and Action: Interweaving 27 Flavors of
Action Research. Action Research, 1(2), 133–152.
doi:10.1177/14767503030012002

30. 30
References
• Collins, A., Joseph, D., & Bielaczyc, K. (2004).
Design Research: Theoretical and Methodological
Issues. The Journal of Learning Sciences, 13(1),
15–42. Retrieved from
http://www.jstor.org/stable/1466931 .
• Friedman, V. J., & Rogers, T. (2009). There is
Nothing so Theoretical as Good Action Research.
Action Research, 7(1), 31–47.
doi:10.1177/1476750308099596
• Goodyear, P. (2005). Educational Design and
Networked Learning: Patterns, Pattern Languages
and Design Practice. Australasian Journal of
Educational Technology. Retrieved from
http://www.ascilite.org.au/ajet/ajet21/goodyear.html

31. 31
References
• Herrington, J., & Oliver, R. (1999). Using Situated
Learning and Multimedia to Investigate HigherOrder Thinking. Journal of Educational Multimedia
and Hypermedia, 8(4), 401–422. Retrieved from
http://www.aace.org/pubs/jemh/v8n4.htm
• Herrington, J., & Oliver, R. (2000). An Instructional
Design Framework for Authentic Learning
Environments. Educational Technology Research
and Development, 48(3), 23–48.
doi:10.1007/BF02319856
• Hodgkinson, H. L. (1957). Action Resarch - A
Critique. Journal of Educational Sociology, 31(4),
137–153. Retrieved from
http://www.jstor.org/stable/2264741

32. 32
References
• Laurillard, D. (2008). The Teacher as Action
Researcher: Using Technology to Capture
Pedagogic Form. Studies in Higher Education,
33(2), 139–154. Retrieved from
http://dx.doi.org/10.1080/03075070801915908
• Oh, E., & Reeves, T. (2008). Design Research vs.
Instructional Systems Design: Implications for
Educational Technologists. In World Conference on
Educational Multimedia, Hypermedia and
Telecommunications 2008 (pp. 2119–2127). Vienna,
Austria: AACE. Retrieved from
http://www.editlib.org/index.cfm?fuseaction=Reader.
ViewAbstract&paper_id=28664

33. 33
References
• Reeves, T. C. (2000). Enhancing the Worth of
Instructional Technology Research through “Design
Experiments” and Other Development Research
Strategies. In Annual Meeting of the American
Educational Research Association (pp. 1–15). New
Orleans, LA, USA. Retrieved from
http://it.coe.uga.edu/~treeves/
• Sandoval, W. A. (2004). Developing Learning
Theory by Refining Conjectures Embodied in
Educational Designs. Educational Psychologist,
39(4), 213–223. doi:10.1207/s15326985ep3904_3