This presentation was co-authored by Tim Coughlan (Nottingham), Beck Pitt (OU), Patrick McAndrew (OU) and Nassim Ebrahimi (Anne Arundel). It was presented at OER13, Nottingham, UK which took place 26-27 March 2013.

1. Assessing OER impact across varied organisations
and learners: experiences from the Bridge to
Success initiative
Tim Coughlan
Horizon Digital Economy Research
& School of Computer Science,
The University of Nottingham (UK)
Patrick McAndrew & Beck Pitt
Institute of Educational Technology,
The Open University (UK)
Nassim Ebrahimi
Anne Arundel Community
College, MD, USA.

2. Overview and Aims
• Introducing the Topic
• Introducing Bridge to Success
• Research Methods Used
• Overview of Users and Pilot
Contexts
• Definitions of “Impact”
• Issues in Assessing “Impact”…
• Towards an Effective General
Model for Research

3. Introducing the Topic
• How to understand the impact of OER and of initiatives?
• What is meant by impact?
• What do different stakeholders want to know?
• What are the particular complexities of research in this space?
• How could we overcome them?

4. What is Bridge to Success?
Next Generation
Learning Challenges
The project aimed to offer free, open
educational resources to prepare adults to
successfully and confidently transition to a
college environment in the US, to pursue
advanced qualifications, or to be successful
in their chosen careers…

5. Research Methods
“What is the impact of Bridge
to Success on student success
and educator practice?”
Product Design/Adaptation
Process
Contexts of Use
Student and
Institutional Experience
Student Outcomes
Project Impact
Evaluation Research
Surveys
Quizzes
Interviews & Focus Groups
Pre- and Post-Surveys
Analysis of interviews, meetings,
emails etc.
Courses underwent accessibility,
usability and design analysis
Action Research
Periodic Interviews with key
stakeholders and the project
team
Quantitative
Analytics of website use
Institutional student data

6. Overview of Users and Pilots
Who piloted the content?
• A total of 16 US college and non-college institutions (October
2011 – December 2012).
• As at August 2012 a total of 17 out of 26 completed pilots had
been conducted with cohorts of low-income students. A further
16 pilots were ongoing at this date (total = 42 distinct pilots).
• As at February 2013 9 US college and non-college institutions
were continuing to utilise Bridge to Success materials, in a variety
of contexts.

7. Defining “Impact”
Differences across Stakeholders
• Funders
• Policy Makers
• Educational Organisations
• Academic Research Collaboration

8. Defining “Impact”
Measures of Impact:
• Retention / Completion / Persistence.
• Whether a student has “mastered” material.
• “Deeper Learning.”
• Changes to educator practice.
• Increased learner confidence.
Collaboration

9. Assessing “Impact” in Community Colleges
• Nine colleges piloted Bridge to Success materials during the
project’s pilot phase (October 2011 – August 2012).
As at August 2012
• Of the completed pilots that have submitted 100% of their data,
and as reported by instructors, 88% of students who completed B2S
materials persisted to next semester.
• 98% of these students mastered the subject matter and 96%
mastered deeper learning.

10. Assessing “impact” in non-college institutions
EXAMPLE ONE
International Workforce Development Agency: supported people across Maryland
for over 90 years.
This pilot utilised specific units of the B2S maths course until end of August 2012 to
enable students to complete a new requirement math pre-test for a Weatherization
program.
Students who originally failed the math entrance exam, 80% passed after working
with B2S resources for a 1-3 week period.

11. Assessing “impact” in non-college institutions
EXAMPLE TWO
Family Support Centre in residential area where:
“85% of families are headed by a single female parent;
90% have not completed high school and do not have a GED;
95% are unemployed, underemployed or receive welfare or other social services.”
(Source: Case Study Two Family Support Centre leaflet 2012)
Piloting both Bridge to Success courses to support preparation for General
Education Development (GED) and pre-GED examinations.
Outcome of Spring 2012 pilot with 8 participants:
•2 obtained their GED certificate,
•2 less advanced learners referred to another programme,
•2 students sat their GED examinations but need to retake,
•1 participants is now employed following a job training programme and
will take their GED during March 2013,
•1 dropped out of Waverly programme.

12. Use of B2S materials to date
Between 1 October 2011 – 17 March 2013:
147,183 Page Views
18,591 Unique Visitors

13. How should we really measure impact?
• It is inherent in the Open concept that resources are used in
different contexts and in varied ways.
• This is part of what makes for great impact
• But it makes it complex to holistically measure or compare the
value of an OER or an OER initiative

14. Research Methods and Outcomes
• Usage Statistics
• Show broad evidence of amount of use, trends, locations of users
• Can show paths of individual users / identified groups
• No data on why many events happen (e.g. did a teacher tell them to stop
at that point or did they get stuck?)
• In-built pre / post unit assessments
• Show evidence of improvement in scores
• But only in a sample who took both quizzes

15. Research Methods and Outcomes
• Observations and ethnographic work
• Provide evidence of impact outside the expected
• Begin to understand why it works or where it doesn’t
• Time consuming, reliant on opportunities, doesn’t scale well
• Institutional Data
• Longer-term, detailed understanding of impact on individuals
• Heterogeneous
• Requires long-term involvement and relationship building
• Can support institutional comparison, so may be wary of sharing
• Hard to find suitable comparator if targeting at (e.g. underachievers)

16. Research Methods and Outcomes
• Pre / post course surveys
• Can be used to identify learner and their context in the system and link with other
data (e.g. institution ID)
• End point of use is hard to know, and unlikely to be when a learner wants to fill in
a survey
• Creates an initial barrier to getting on with using the resources
• Surveys / Interviews with Instructors
• Understand context of learning and use of OER
• Elicit instructors judgement on types of impact (e.g. ‘Deeper Learning’ definitions)
• Can be combined with support and advice
• Scope for definitions to be interpreted differently

17. What is required?
• Understanding the different contexts of learners
• Identifying individuals and learning contexts across data sources
• Operationalising varied impact definitions to be studied at scale
• Being able to adapt and expand to opportunistic assessments

18. An Effective General Model

19. Conclusions and Next Steps
• Broadness of use should be acknowledged in OER research
• Methodological and technical innovation needed
• Mixed methods approaches
• Relationship building and long term planning
• More research on use of Bridge to Success content via the
Hewlett funded OER Research Hub project
• Reversioning of the courses to take into account user feedback
and analysis

20. THANK YOU!
Tim Coughlan tim.coughlan@nottingham.ac.uk
@t1mc
Beck Pitt Beck.Pitt@open.ac.uk
@BeckPitt
Patrick McAndrew p.mcandrew@open.ac.uk
@openpad
Nassim Ebrahimi nebrahimi@aacc.edu

21. Resources
• Watch a review of Bridge to Success functionality and the
courses: https://www.youtube.com/watch?v=9pHyYO5d01I
• Visit the Bridge to Success website:
http://bridge2success.aacc.edu/
• Visit the OER Research Hub website: http://oerresearchhub.org/

22. Bibliography
• Atkins, D. E., Seely Brown, J., Hammond, A. L., A Review of the Open
Educational Resources Movement: Achievements, Challenges and New
Opportunities (2007)
(http://www.hewlett.org/uploads/files/Hewlett_OER_report.pdf)
• Bailey & Cho Issue Brief: Developmental Education in Community Colleges
prepared for The White House Summit on Community Colleges (October 2010)
• Bridge to Success Report One: Adaptation, Integration and Engagement
• Case Study Two Family Support Centre Leaflet 2012
• College Board: Advocacy & Policy Centre Education Pays 2010
(http://trends.collegeboard.org/sites/default/files/education-pays-2010-full-report.
pdf)
• College Board: Advocacy & Policy Centre Trends in Student Aid 2012
(http://trends.collegeboard.org/sites/default/files/student-aid-2012-full-report-
130201.pdf)
• Kaminski, K. Seel, P. & Cullen K. Technology Literate Students? Results from a
Survey (2003, EDUCAUSE Quarterly, No 3)