Barbour, M. K. (2011, April). An introduction to a new research paradigm: Design-based research. An invited presentation to the National Centre for Teaching and Learning at Massey University, Palmerston North, New Zealand.

1. An Introduction to a New
Research Paradigm: ‘Design-
Based Research’
Michael K. Barbour
Assistant Professor
Wayne State University

2. It all began last spring when I read two queries from doctoral students on the
Qualitative Research for the Human Sciences listserv. Both students came from
large public institutions of higher education, one in the USA and the other in
Canada. The first student wrote that she intended to focus her dissertation
research on the quality of "discourse" that takes place in cafes and coffee shops
located inside bookstores. She complained that she had found no "literature" on
this topic and asked the listserv participants for some guidance. The second
student announced that he was preparing a dissertation prospectus centered on the
question of how people learned about opportunities to take SCUBA diving
lessons and what motivated them to register for such courses. He also sought
directions to relevant literature and advice from the listserv membership.
After pondering these queries, I posted a message asking whether faculty
members at taxpayer-supported universities have a moral responsibility to guide
their students toward "socially responsible" research questions. In my posting, I
suggested that in the face of problems such as adult illiteracy, attacks on public
education, "at-risk" students, homelessness, AIDS, and the like, faculty members
should attempt to inspire in students a dedication to research that would "make a
difference."
Thomas Reeves, University of Georgia
Peter Dean Lecture at the 1995 Association for Educational Communications

3. • Vast resources
going into
education research
are wasted
• We [educational
researchers]
employ weak
research methods,
write turgid prose,
and issue

4. • Too much useless
work is done
under the
banner of
qualitative
research
• Qualitative research…. [yields]
….little that can be generalized
beyond the classrooms in
which it is conducted

5. Problems with Educational
Technology Research
1.Misunderstanding about basic and applied
research.
2.Poor quality of educational technology
research.
3.Disappointing research synthesis.

6. Problems with Educational
Technology Research
1.Misunderstanding about basic and applied
research.
2.Poor quality of educational technology
research.
3.Disappointing research synthesis.

7. Basic and Applied Research
• Basic (aka fundamental or pure)
research is driven by a scientist's
curiosity or interest in a scientific
question. The main motivation is
to expand man's knowledge,
not to create or invent something.
• Applied research is designed to
solve practical problems of the
modern world, rather than to
acquire knowledge for
knowledge's sake. One might say
that the goal of the applied
scientist is to improve the
human condition.

8. Basic and Applied Research

9. Problems with Educational
Technology Research
1.Misunderstanding about basic and applied
research.
2.Poor quality of educational technology
research.
3.Disappointing research synthesis.

10. Poor Quality of Educational
Technology Research
“The best current evidence is that
media are mere vehicles that
deliver instruction but do not
influence student achievement
any more than the truck that
delivers our groceries causes
changes in our nutrition.”
Richard Clark
Review of Educational Research
1983

11. • Abundant technology
has not led to extensive
use of computers for
“tradition-altering
classroom instruction.”
• The small percentage of
computer-using
instructors only use it to
maintain existing
classroom practices.

12. What Does the Secondary E-Learning
Research Say About Student Performance?

13. Student Performance
• performance of virtual
and classroom students
in Alberta were similar
in English and Social
Studies courses, but
that classroom students
performed better
overall in all other
subject areas (Ballas &
Belyk, 2000)

14. Student Performance
• over half of the students who
completed FLVS courses
scored an A in their course and
only 7% received a failing
grade (Bigbie & McCarroll,
2000)
• students in the six virtual
schools in three different
provinces performed no worse
than the students from the
three conventional schools
(Barker & Wendel, 2001)

15. Student Performance
• FLVS students performed
better on a non-mandatory
assessment tool than students
from the traditional classroom
(Cavanaugh et al., 2005)
• FLVS students performed
better on an assessment of
algebraic understanding than
their classroom counterparts
(McLeod et al., 2005)

16. Let’s look a little closer...

17. Students and Student Performance
Ballas & performance of virtual and participation rate in the
Belyk, 2000 classroom students similar assessment among virtual
in English & Social Studies students ranged from 65% to
courses, but classroom 75% compared to 90% to
students performed better 96% for the classroom-based
in all other subject areas students
Bigbie & over half of the students between 25% and 50% of
McCarroll, who completed FLVS students had dropped out
2000 courses scored an A in of their FLVS courses over
their course and only 7% the previous two-year
received a failing grade period

18. Students and Student Performance
Cavanaugh et FLVS students performed speculated that the virtual
al., 2005 better on a non- school students who did
mandatory assessment take the assessment may
tool than students from have been more
the traditional classroom academically motivated and
naturally higher achieving
students
McLeod et FLVS students performed results of the student
al., 2005 better on an assessment performance were due to
of algebraic understanding the high dropout rate in
than their classroom virtual school courses
counterparts

19. Student Performance and Students
But are we really
comparing apples to
apples?

20. The Students
• the vast majority of VHS
Global Consortium students
in their courses were
planning to attend a
four-year college (Kozma,
Zucker & Espinoza, 1998)
• “VHS courses are
predominantly designated as
‘honors,’ and students
enrolled are mostly college
bound” (Espinoza et al., 1999)

21. The Students
The preferred characteristics
include the highly motivated,
self-directed, self-disciplined,
independent learner who
could read and write well,
and who also had a strong
interest in or ability with
technology (Haughey &
Muirhead, 1999)

22. The Students
• “only students with a high
need to control and structure
their own learning may
choose distance formats
freely” (Roblyer & Elbaum,
2000)
• IVHS students were “highly
motivated, high achieving,
self-directed and/or who liked
to work independently” (Clark
et al., 2002)

23. The Students
• the typical online student
was an A or B student
(Mills, 2003)
• 45% of the students who
participated in e-learning
opportunities in Michigan
were “either advanced
placement or
academically advanced”
students (Watkins, 2005)

24. Problems with Educational
Technology Research
1.Misunderstanding about basic and applied
research.
2.Poor quality of educational technology
research.
3.Disappointing research synthesis.

25. Disappointing Research Synthesis
Kannapel and DeYoung (1999) found that rural schools
contained a strong sense of community and were
regularly the focus of the cultural and social aspects of
the community. They also found that “extracurricular
and non-academic activities are often valued as much or
more than academics, and a higher proportion of
students participate in extracurricular activities than in
urban schools” (p. 170).

26. Disappointing Research Synthesis
In their review of key literature over the past 25 years,
Kannapel and DeYoung (1999) found that rural schools
contained a strong sense of community and were
regularly the focus of the cultural and social aspects of
the community. They also found that “extracurricular
and non-academic activities are often valued as much or
more than academics, and a higher proportion of
students participate in extracurricular activities than in
urban schools” (p. 170).

27. Disappointing Research Synthesis
In their review of key literature over the past 25 years,
Kannapel and DeYoung (1999) found that rural schools
contained a strong sense of community and were
regularly the focus of the cultural and social aspects of
the community. They also found that “extracurricular
and non-academic activities are often valued as much or
more than academics, and a higher proportion of
students participate in extracurricular activities than in
urban schools” (p. 170). It should be noted that three of
the eleven “key” pieces of literature were written by
DeYoung, one of the two authors of this review.

30. QuickTime™ and a
decompressor
are needed to see this picture.

31. Examining Effect Sizes
Teacher
Effects Zone of
Desired Effects
Developmental
Effects
Reverse
Effects

32. Primary & Secondary E-Learning
Meta-Analysis
• Cavanaugh (2001) - 16 studies
– +0.147 in favor of K-12 distance education
• Cavanaugh et al. (2004) - 14 studies
– -0.028 for K-12 distance education
• Means et al. (2009) - 46 studies (5 on K-12)
– +0.24 favoring online over face-to-face*
– +0.35 favoring blended over face-to-face*

33. Results of Interest
• Second and third chance programs (d=0.50)
• Matching style of learning (d=0.40)
• Computer assisted instruction (d=0.37)
• Decreasing disruptive behavior (d=0.34)
• Programmed instruction (d=0.24)
• Individualized instruction (d=0.23)
• Class size (d=0.21)
• Charter schools (d=0.20)
• Web-based learning (d=0.18)
• Home-school programs (d=0.16)
• Teacher training (d=0.11)
• Teacher subject matter knowledge (d=0.09)
• Distance education (d=0.09)
• Student control over learning (d=0.04)
33
33

34. Results to Consider
• Providing formative evaluation (d=0.90)
• Micro teaching (d=0.88)
• Teacher clarity (d=0.75)
• Providing feedback (d=0.73)
• Teacher-student relationships (d=0.72)
• Teaching strategies (d=0.60)
• Cooperative vs. individualistic learning (d=0.59)
• Study skills (d=0.59)
• Direct instruction (d=0.59)
• Mastery learning (d=0.58)
• Worked examples (d=0.57)
• Concept mapping (d=0.57)
• Goals (d=0.56)
• Peer tutoring (d=0.55)
• Cooperative vs. competitive learning (d=0.54)

35. Maybe The Problem Is How We
Conduct Research?

36. Randomized Controlled Trials
1. Is there a control group?
2. Are the control and
experimental groups
assigned randomly?
3. If it is a matched study,
are the groups extremely
similar?
4. Is the sample size large enough?
5. Are the results statistically significant?

37. http://ies.ed.gov/ncee/wwc/

38. What Works Clearinghouse
Ironically, the WWC personnel have been able to identify
very few educational programs and practices that have the
evidence that is sufficiently rigorous according to their
own criteria to warrant their inclusion in the What Works
database. For example, a review of over 1,300 studies that
examined the effect of teacher professional development
on student achievement found that only nine met WWC
standards for rigorous evidence (
Yoon, Duncan, Lee, Scarloss, & Shapley, 2007).
Can Educational Research Be Both Rigorous and Relevant?Thomas Reeves,
2011
http://www.educationaldesigner.org/ed/volume1/issue4/article13/index.htm

39. QuickTime™ and a
decompressor
are needed to see this picture.

40. QuickTime™ and a
decompressor
are needed to see this picture.

41. More Problematic Research
Online 7 principles of Interviews with teachers and course
Course effective online developers at a single virtual school,
Design course content with no verification of whether the
for adolescent interviewees’ perceptions were actually
Barbour learners effective (or any student input for that
(2005; 2007) matter)
Online 37 best Interviews with teachers at a single
Teaching practices in virtual school selected by the virtual
asynchronous school itself. Their teachers’ beliefs
DiPietro et online teaching were not validated through observation
al. (2008) of the teaching or student performance.

42. There Must Be A Better Way!

43. Design-Based Research
Reeves (2006)

44. Design-Based Research
• an emphasis on
conducting research in
authentic, natural
educational contexts,
rather than laboratories
• the desire for research to
have a practical impact, by
having clear relevance for
the improvement of
education
(Walker, nd)

45. Design-Based Research
• an insistence on theory-guided
educational intervention:
‘interventions embody specific
theoretical claims about teaching
and learning’
• a pluralist approach with respect
to theories, research designs,
methods, and procedures -
includes mixed methods where
qualitative and quantitative
techniques are used in
combination
(Walker, nd)

46. Design-Based Research
• the use of an iterative
design and evaluation
cycle: interventions are
adjusted as the research
proceeds
• a focus on how the
intervention worked
(Walker, nd)

48. Virtual High School Global Consortium
• first annual evaluation
– Kozma, Zucker &
Espinoza, 1998
• focused specifically on
the seven goals set by
VHS
• identified five areas to
focus on for future
practice

49. Virtual High School Global Consortium
• second annual
evaluation
– Espinoza, Dove, Zucker
& Kozma, 1999
• again focused
specifically on the
seven goals set by VHS
• identified three areas
to focus on for future
practice

50. Virtual High School Global Consortium
• third annual evaluation
– Kozma, Zucker,
Espinoza, McGhee,
Yarnall & Zalles, 2000
• re-examined status of
last year’s evaluation
finding
• focused upon only one
of the seven goals set
by VHS

51. Virtual High School Global Consortium
• content-specific
investigations
– Yamashiro & Zucker, 1999
• examined quality of
netcourses offered by
VHS
• developed standards for
future course
development

52. Virtual High School Global Consortium
• content-specific
investigations
– Elbaum, McIntyre &
Smith, 2002
• seventeen essential
elements for online
teaching
• written by VHS staff

53. Virtual High School Global Consortium
• final evaluation
– Zucker & Kozma, 2003
• examined students,
teachers,
administrators
perceptions of the
program
• outlined successes and
areas to focus on for
future years

54. Your
Questions
and
Comments

55. Assistant Professor
Wayne State University, USA
mkbarbour@gmail.com
http://www.michaelbarbour.com
http://virtualschooling.wordpress.com