T.J. Bliss, Lane Fischer, John Hilton III, T. Jared Robinson, David Wiley Brighma Young University

1. Assessing the Educational Value of OER
The Open Education Group
http://openedgroup.org/
• T.J. Bliss
• Lane Fischer
• John Hilton III
• T. Jared Robinson
• David Wiley

2. Recent Studies
• Scottsdale Community College
• Houston Community College
• Virginia State University
• Kaleidoscope Open Course Initiative
• Mercy College
• Nebo School District

3. Houston Community College
(paper available)
• Houston Community College
(HCC) is a large community
college with more than
70,000 students. 33% of its
students are Hispanic, 33%
are African American, 17%
are white, 14% Asian, and 3%
are classified as ‘other.’ Fifty-
nine per cent of the students
are female, and 41% are
male.

4. Textbook Adoption
• In 2011 HCC’s
Psychology
department chose to
adopt Flat World
Knowledge’s (FWK)
textbook Introduction
to Psychology.
• During a pilot study 690
students used the FWK
text.
• So what happened?

5. HCC – Student Feedback
• 84% of students surveyed agreed with the
statement that “Having a free online book helps
me go to college.”
• Of the 108 students who responded to a
question regarding the difficulty of using an
online text, 45 (42%) said that it was easy, 28
(26%) said that it was moderately easy, 26 (24%)
said that it was neither hard nor easy, and nine
(8%) said that it was moderately difficult.

6. HCC – Results Comparing Students from two
instructors who taught in Spring 2011 and Fall
2011

7. Virginia State University
(paper available)
• Virginia State University
(VSU) has 5,300 students.
They offer 55 baccalaureate
and masters degree
programs, two doctoral
degree programs and three
certificates.
• VSU’s Business Department
adopted nine FWK texts and
paid $30.00 per student for
an unlimited seat license for
all FWK content.

8. VSU – Student Feedback from 148 students

9. VSU – Results
• 95% of students accessed at least some of FWK
resources (in contrast with 47% of students
purchasing textbooks one year previously).
• Students downloaded a variety of resources

10. VSU – Results
In total there were seven courses that used FWK textbooks (COBU101, COBU200, COBU201, COBU210,
COBU300, COBU301, COBU302, COBU310, MISY350), and ten courses that did not use FWK textbooks
(COBU110, COBU111, COBU155, COBU170, COBU202, COBU260, COBU304, COBU342, COBU343, COBU400).
While these were clearly different courses, they were roughly equivalent in terms of course difficulty.
ABC_ in
Core
Course
s
using
FWK
DFW_
in
core
courses
using
FWK
Total DFW %
with
FWK
ABC_
in
core
courses
w/o
FWK
DFW_
in
core
courses
w/o
FWK
Total DFW %
not
using
FWK
Fall
2010
611 151 762 24.7%830 270 1100 32.5%
Spring
2011
531 100 631 18.8%834 242 1076 29.0%

11. Kaleidoscope Open Course Initiative
(paper available)
Kaleidoscope Open Course
Initiative
(http://www.project-
kaleidoscope.org/) is an open
education initiative initially
comprised of eight
community colleges and open
access four-year institutions.
These colleges work together
to create courses that replace
traditional, textbooks with
OER.

12. Kaleidoscope Open Course Initiative
During the 2011-2012
academic
year, Kaleidoscope Open
Course Initiative impacted
about 4,000 students
across 80 teachers in eight
institutions.

13. Kaleidoscope Open Course Initiative
• 58 teachers from 8
PK institutions
completed at least
some items on a
questionnaire.
• 490 students
completed at least
some items on a
questionnaire.

14. Kaleidoscope Open Course Initiative:
Cost
Instructor reports of
textbook costs during
Kaleidoscope
Instructor reports of
textbook costs pre-
Kaleidoscope

15. Total number of students: 14,606
Kscope 3,876
Non-kscope 10,739
Quantifying Textbook Costs

17. • Average book cost per class section:
$92.90
• Total potentially spent by non-
kaleidoscope students: $997,653.10
• Total potentially saved by
kaleidoscope students: $360,080.40
• Total potential savings:
$1,357,733.50

18. Kaleidoscope Open Course Initiative – Teacher
Perceptions

19. Teachers: How do the OER textbooks
compare with other textbooks you
have used?
0
10
20
30
40
50
60
Worse The Same Better
Percentofteachers
Teacher perceptions of quality

20. Students: How do the OER textbooks
compare with other textbooks you
have used?
0
10
20
30
40
50
60
Worse The Same Better
Student perceptions of text quality
Percentofstudents

21. Kaleidoscope Open Course Initiative
– Student Perceptions
• 160 students provided a description of what
they thought made the OER texts better.
Their responses clustered in six major
categories: technical advantages
(8%), learning aides (9%), customization
(10%), cost (20%), access (26%), and
quality/presentation (27%).

22. Kaleidoscope Open Course Initiative
– Student Perceptions
• Conversely, 20 students provided a
description of what they thought made the
OER texts worse than other texts they had
used in the past. Their responses clustered
in 2 major categories; six students described
technology issues and 15 students took
issue with text quality.

23. 0
5
10
15
20
25
30
35
40
45
50
Subjects students commented on
Numberofstudentcomments

24. Mercy College Math
About Mercy: Mercy College is
dedicated to making a high-quality
education attainable offering more
than 90 undergraduate and
graduate programs within its five
schools: Business, Education, Health
and Natural Sciences, Liberal
Arts, and Social and Behavioral
Sciences. The New York
metropolitan area college of nearly
10,000 students has four campus
locations including: Dobbs
Ferry, Bronx, Manhattan, and
Yorktown Heights and offers more
than 25 degree programs online.

25. No OER All OER
Math116 Fall 2011 Fall 2012
TOTAL STUDENTS 697 695
Pass (A-C) 443 479
Not Pass (D/F/FW) 254 216
% Pass 63.60% 68.90%
% Not Pass 36.40% 31.10%
Mercy College Math

26. Nebo School District
The Nebo School District is
a public school district
located about one hour
south of Salt Lake City. It
serves 40 schools and
30,500 students.

27. Nebo School District
Does the adoption of open
science textbooks significantly
affect science learning outcomes
for secondary students?

28. • 4,137 students took the
state Criterion Referenced
Tests (CRTs) in earth
systems, biology, or
chemistry. (Excludes AP
sections, special
education sections, and
bio-agriculture sections)
• 2,350 students used
traditional textbooks,
while 1,787 used OER
textbooks.

29. Method
• Quasi-experimental design with
– Treatment and control groups
– Groups matched by propensity score
• Dependent variable: 2012 science CRT score
• Independent variable: Textbook condition
• Covariates: age, gender, special education,
English language proficiency, 2011 test data,
2011 GPA, race, and free and reduced lunch

30. Matching
• Propensity Score Matching
• Based on the conditional probability of
assignment to treatment condition, given the
covariates
• Increased group balance by 98%
• Approximates random assignment

31. Results

32. Summary
• Houston Community College – Exploring a
variety of other OER.
• Virginia State University – Continues to use
FWK texts.
• Kaleidoscope Open Course Initiative – 28
schools
• Mercy Math – Expanding work with OER.
• Nebo School District – Utah has approved the
textbooks for statewide use; preliminary
surveys suggest up to 75,000 students will use
the open textbooks.

33. Questions?
John Hilton III
http://johnhiltoniii.org
http://openedgroup.org/