It can be hard to demonstrate the impact of open education. The Open Education Research Hub (OER Hub, n.d.) is based at The Open University, UK. The Hub has been identified as a potential exemplary case for REF2021 (n.d.), the quality assessment exercise for UK universities. This presentation will explain the work of OER Hub by providing an overview of research activity alongside an impact narrative. Delegates will be provided with strategies for conceptualising and providing evidence of OER impact in their own practice. The sequence begins in 2012 with a range of international research collaborations and fellowships (de los Arcos, 2014; Farrow, 2015) and continued through further waves of funded work conducted by the same team. Over time the OER Hub portfolio has evolved to include open textbooks (Pitt, 2015); open business models (Orr et al., 2018); MOOC (BizMOOC 2018; Pitt et al., 2017); supporting open researchers (de los Arcos et al., 2016; Jordan & Weller, 2017); participatory action research (Farrow et al., 2016) and providing framing narratives (Weller, 2014) as well as continuing to evaluate OER impact for a range of national and international funders. Open practice is at the core of our achievements. Through our research we have discovered that open research processes can be highly effective ways of aligning activity to stakeholder needs. We have identified five key pathways to impact for our case: • Contributing to a high-quality global evidence base OER and its impact • Raising profile and understanding of OER widely • Building an inclusive, global community which influences practice • Promoting open research and open values • Influence on policy and consultation https://oer19.oerconf.org/sessions/shaping-open-practice-oer-research-impact-for-research-excellence-framework-2021-o-117/

1. Shaping Open Practice: OER Research
Impact for Research Excellence
Framework (2021)
#OER19
Galway, Republic of Ireland 10th April 2019
Dr. Rob Farrow
Dr. Beck Pitt
Prof. Martin Weller
Additional contributions:
Prof. Jane Seale; Dr. Katy Jordan

2. 2
01 What is REF 2021?
A quick start guide to the Research Excellence
Framework
02 Openness and Impact
Is openness an impediment to impact?
03 What have we done?
The sequence of activities we wish to report on
04 Documenting Impact / Examples
How our case is being put together
05 Pathways to Impact
The five routes from openness we have identified
06 Advice for the REF 2021
Practical tips for your research submission(s)
STRUCTURE

3. What is REF 2021
A guide to the Research Excellence Framework

4. 4
COMPARISON WITH EARLIER VERSIONS OF THE EXERCISE
WHAT IS REF 2021?
Research Assessment Exercise (RAE) ran
from 1992 – 2008. It used a four (or six)
point scale to judge the quality of research
outputs according to “originality,
significance and rigour”.
The REF was first carried out in 2014,
replacing the previous Research
Assessment Exercise. The REF is a
process of expert review, carried out by
expert subject panels.
REF 2021 is less of a departure from form
than REF2014, but has aims at more
consistent eligibility for submissions and
has a more diverse lens for ‘impact’.

5. 5
CRITICAL VOICES
REF 2021
https://www.theguardian.com/education/201
8/dec/05/our-research-funding-system-is-
shortchanging-the-humanities
https://www.theguardian.com/higher-
education-network/2016/oct/17/why-is-so-
much-research-dodgy-blame-the-research-
excellence-framework
STERN REPORT (2016)
COST OF REF 2014 = £246 million
GAMIFICATION: Institutions hire staff selectively to
improve their REF returns
SELECTION: Being selective about reported outputs
tends to favour white male researchers. ‘Risky’ or
unconventional approaches de-emphasized
Impact on teaching not recognised; industry under-
recognised
Multi-disciplinary projects, collaborations discouraged
https://assets.publishing.service.gov.uk/government/uploads/system/uplo
ads/attachment_data/file/541338/ind-16-9-ref-stern-review.pdf

6. 6
WHAT IS DIFFERENT THIS TIME?
WHAT IS REF 2021?
REF2021 rule change: previous REF asked for 4 publications per person. Some
people were not included for fear of not having 4 publications of sufficient quality. For
2021 everyone is considered research active and will submit 1-5 papers.
Papers can be included as long as they are “in the public domain”. Pre-prints and pre-
publication online are valid.
3* and 4* papers will be rewarded with funding.
There is a lot of flexibility about what can be submitted: journal papers; book chapters;
monographs; conference papers; project reports; etc.
Teaching is now considered a practice, so impact on teaching is now valid
It is anticipated that there will be observed an increase in 4* papers in REF2021 as
academics become better at ameliorating their work to meet the demands of the REF.
https://www.ref.ac.uk/media/1092/ref-2019_01-guidance-on-submissions.pdf

7. 7
WHAT IS DIFFERENT THIS TIME?
WHAT IS REF 2021?
REF2021 rule change: previous REF asked for 4 publications per person. Some
people were not included for fear of not having 4 publications of sufficient quality. For
2021 everyone is considered research active and will submit 1-5 papers.
Papers can be included as long as they are “in the public domain”. Pre-prints and pre-
publication online are valid.
3* and 4* papers will be rewarded with funding.
There is a lot of flexibility about what can be submitted: journal papers; book chapters;
monographs; conference papers; project reports; etc.
Teaching is now considered a practice, so impact on teaching is now valid
It is anticipated that there will be observed an increase in 4* papers in REF2021 as
academics become better at ameliorating their work to meet the demands of the REF.
https://www.ref.ac.uk/media/1092/ref-2019_01-guidance-on-submissions.pdf

8. 8
WHAT IS DIFFERENT THIS TIME?
WHAT IS REF 2021?
1. Case study fields remain but with metadata
2. Underpinning research must be 2* and published since 2000
3. Only impacts from Aug 2013 can be considered
4. Minimum 2 cases up to 20 FTE then 1 per 15 FTE
5. Impacts from a body of research must demonstrate substantive
institutional contribution
6. Pedagogical impacts can include home institution
7. Public engagement is not a pathway to impact
8. All evidence submitted with case studies must be independently
verifiable and based on evidence (not opinion)
9. Continuation case studies are where additional impacts have arisen in
current REF window from same underpinning research as REF 2014

9. 9
WHAT IS DIFFERENT THIS TIME?
WHAT IS REF 2021?
1. Case study fields remain but with metadata
2. Underpinning research must be 2* and published since 2000
3. Only impacts from Aug 2013 can be considered
4. Minimum 2 cases up to 20 FTE then 1 per 15 FTE
5. Impacts from a body of research must demonstrate substantive
institutional contribution
6. Pedagogical impacts can include home institution
7. Public engagement is not a pathway to impact
8. All evidence submitted with case studies must be independently
verifiable and based on evidence (not opinion)
9. Continuation case studies are where additional impacts have arisen in
current REF window from same underpinning research as REF 2014

10. 10
MEASURES OF QUALITY
WHAT IS REF 2021?
Outputs sub-profile (65%) (originality, significance, rigour)
Impact sub-profile (20%) (reach, significance)
Environment sub-profile (15%) (vitality, sustainability)
Rating Qualities
4* World-leading
3* Internationally excellent
2* Internationally recognised
1* Nationally recognised
(Unclassified) (Not nationally recognised)

11. Openness & Impact
What is the relationship? Does it enable or impede

12. 14
ADVANTAGE OR IMPEDIMENT?
OPENNESS AND IMPACT
DRIVER BARRIER
Citation rates for OA Prestige of OA platforms /journals has
historically been considered problematic
(though this is changing with Green OA
required)
Visibility, presence How can we objectively demonstrate
influence on practice? Significance?
Indirect impact through commons
Stakeholder feedback loops through open
practice: blogging, social media
Working with one or two key stakeholders
might be easier
Encourage re-use of outputs through open
licence
How can we track this re-use? Can we
anticipate unexpected forms of reuse
What kind of time delay can we expect?
Improved “reach” Global reach is good, but keeping track of
all our stakeholders becomes more
challenging

13. What have we done?
The sequence we are reporting on

14. 16
-2012 ’UNDERPINNING RESEARCH’
OPEN EDUCATION RESEARCH HUB
Open Learning Network was an international
research and fellowship network for
aggregating, sharing, debating and improving
Open Educational Resources (OER).
The Evidence Hub for Open Education was
a Collective Intelligence tool developed as
part of the OLnet project.
Global evidence collection around the
impact of OER; organised around key
hypotheses

15. 17
2012-2015 OER RESEARCH HUB
OPEN EDUCATION RESEARCH HUB
A range of international research collaborations and fellowships synchronised by key themes and education sectors
Keyword Hypothesis
Performance OER improve student performance/satisfaction
Openness People use OER differently from other online materials
Access OER widen participation in education
Retention OER can help at-risk learners to finish their studies
Reflection OER use leads educators to reflect on their practice
Finance OER adoption brings financial benefits for students/institutions
Indicators Informal learners use a variety of indicators when selecting OER
Support Informal learners develop their own forms of study support
Transition OER support informal learners in moving to formal study
Policy OER use encourages institutions to change their policies
Assessment Informal assessments motivate learners using OER

16. Project Co-PILOT

18. OER Evidence Report 2014
OER Data Report 2015
http://oerhub.net/reports/
http://oerhub.net/research-outputs/data/

19. 21
SUSTAINABILITY PHASE (2015-)
OPEN EDUCATION RESEARCH HUB
• Building capacity in the OER research domain
• Conducting research into open education and OER
• Producing resources for the open education research
community
• Acting as champions for open practices in research and
scholarship

20. 2012
• OER Research Hub:
Collaboration across
School/K12; College; Higher
Education & Non-formal
learning
• OER Research Hub
fellowship programme
2013
• OER Research Hub: Collaboration across
School/K12; College; Higher Education &
Non-formal learning
• OER Research Hub fellowship programme
• Bridge to Success
2014
• OER Impact Map
• Evidence report published
• ACE Award for Research
Excellence
• BC Campus Open Textbooks
• P2PU Open Research
• Opening Educational Practice in
Scotland
2015
• Open Education Research Hub
• ExplOERer (Erasmus+)
• OER World Map (II)
• Data report published
• Survey data explorer
• OU Engaging Research award
2016
• Global OER Graduate
Network
• OER World Map (III)
• Open Research Agenda
• Open Research textbook
• Researcher Toolkit
• Twitter 101
• OER Wales Cymru
2017
• OER World Map (IV)
• BizMOOC
• Models for Open, Online, Flexible &
Technology Enhanced (OOFAT)
2018
•OE Global Award for Research Excellence (GO-GN)
•OE Global Award for Open Innovation (OER WM)
•UK Open Textbooks
•GO-GN Diversity Grant
2019
•TIDE Myanmar
•Bringing Learning to Life (England)
•Flexible Essential Skills (Wales)
•European MOOC Consortium
Detail up to 2017: https://www.slideshare.net/OER_Hub/oer-hub-an-overview

21. Documenting Impact
How our case is being put together

22. 25
HOW WE ARE BEING HELPED TO DOCUMENT AND SHAPE OUR IMPACT NARRATIVE
IMPACT: INSTITUTIONAL SUPPORT
It’s very hard to find time for all this
alongside other work! Everyone is
under time pressures and thinking
about a REF submission in 2-3 years
time is hardly a priority.
Some funding has been made
available for a researcher to gather
evidence, collect testimony and be a
prime mover for putting together the
case consistently
We are benefitting from having our
REF 2021 lead review and suggest
direction for our impact cases

23. 26
HOW WE ARE BEING HELPED TO DOCUMET AND SHAPE OUR IMPACT NARRATIVE
IMPACT: INSTITUTIONAL SUPPORT
We are looking for evidence of impact as:
• Activity
• Attitude
• Awareness (e.g. OpenLearn surveys)
• Behaviour
• Capacity
• Opportunity
• Performance
• Policy
• Practice (inc. organisational practices)
• Process
• Understanding
On…
• Audience
• Beneficiary
• Community
• Organisation
• Individuals

24. 27
HOW WE ARE BEING HELPED TO DOCUMET AND SHAPE OUR IMPACT NARRATIVE
IMPACT: INSTITUTIONAL SUPPORT
We are looking for evidence of impact as:
• Activity
• Attitude
• Awareness (e.g. OpenLearn surveys)
• Behaviour
• Capacity
• Opportunity
• Performance
• Policy
• Practice (inc. organisational practices)
• Process
• Understanding
On…
• Audience
• Beneficiary
• Community
• Organisation
• Individuals
The final impact
statement will be just a
couple of sides of A4!

25. Examples of Impact
Some highlights

26. 29
SOME WELCOME RE-USE OF OUR CONTENT
IMPACT: REUSE

27. 30
SOME WELCOME RE-USE OF OUR CONTENT
IMPACT: REUSE

28. 31
SOME WELCOME RE-USE OF OUR CONTENT
IMPACT: REUSE

29. 32
SOME WELCOME RE-USE OF OUR CONTENT
IMPACT: REUSE

30. Pathways to Impact
The five routes from openness we have identified

31. 34
GUIDANCE FROM UK RESEARCH & INNOVATION
PATHWAYS TO IMPACT
https://www.ukri.org/innovation/excellence-with-impact/pathways-to-impact/

32. 35
1. ESTABLISHING AN OPEN EVIDENCE BASE ABOUT OER IMPACT
PATHWAYS TO IMPACT
• Working internationally in open collaboration with a wide
range of partners at various levels of formality
• Sharing outputs openly can massively increase profile &
citation rate
• Tracking re-use of our open data
• Tracking re-use of our open methods, instruments
• Being available to answer queries about outputs
• “Think global; act local”

33. 36
2. RAISING PROFILE AND UNDERSTANDING OF OPEN APPROACHES
PATHWAYS TO IMPACT
• OER Research Hub Fellowships
• Bringing an open element to consortia outside typical
research remit (e.g. BizMOOC, European MOOC
Consortium)
• OOFAT Models project
• Quality assurance, internal training provision at The Open
University (Twitter 101, Open Research)
• MOOC on Open Research (P2PU)
• Input into OU/FutureLearn content (H880)
http://www.open.ac.uk/postgraduate/modules/h880

34. 37
3. AN INCLUSIVE, GLOBAL PRACTICE COMMUNITY
PATHWAYS TO IMPACT
• Global OER Graduate Network
• Knowledge Alliance Projects (BizMOOC; European MOOC
Consortium
• Mentoring and leadership (e.g. SPARC; OpenCon; OE4BW)
• Social Media: presence, consistency, inclusivity, approachability
• Participatory Action Research (Open Research Agenda)
• Consultancies
• Real-life interactions!

35. 38
4. PROMOTING OPEN RESEARCH AND OPEN VALUES
PATHWAYS TO IMPACT
• Global OER Graduate Network
• Open/OA dissemination (papers, data)
• Open Research
• OERRH Ethics Manual
• Ethical framework for open education (Farrow, 2016)
• Reuse of Evidence Hub code (JuxtaLearn, LACE)
• Re-use of research methods, data, etc.

36. 39
5. INFLUENCE ON POLICY AND CONSULTATION
PATHWAYS TO IMPACT
• Evidence to House of Lords Select Committee on Digital
Skills
• Contribution to Creative Commons Policy Workshop
• ALT OER Guide for Policymakers
• Policy Registry on OER World Map
• Opening Educational Practices Scotland Workshops

37. Advice for the REF 2021
Practical tip for your submission(s)

38. 41
RESOURCES
ADVICE FOR REF 2021
Useful guide for conceptualising
impact with advice for practical ways
to put together impact cases.
Further resources:
https://www.fasttrackimpact.com
https://www.fasttrackimpact.com/research-impact-book

39. 42
PRACTICAL STUFF
ADVICE FOR REF 2021
• Journals are likely to be inundated in 2020 as people try to get things
accepted before the deadline.
• It may be worth emailing editors ahead of submission to make sure
that they will be able to publish on time and that the submission is
suitable for their readers.
• Originality and significance are matters of judgement, but rigour is
much easier for assessors to agree over. The highest standards
should be delivered in this respect.
• If you’re providing people with an impact statement you should also
give an opportunity to gather unanticipated impacts – e.g. through a
consistent pro forma. This can also help give a clearer impression of
objectivity.
• Keep an eye out for additional guidance on REF 2021 as mock REFs
are completed and review panels report back

40. 43
PRACTICAL STUFF
ADVICE FOR REF 2021
• Open practices can be an effective way of promoting and recording
impact
• Seek institutional funding or some other resource to support you in
gathering evidence of impact
• It’s been helpful that our work is in an areas considered strategically
important by our institution – maybe there’s something similar out
there for you
• If someone else’s work has influenced your own, tell people about it
and offer an evidence trail through citations, blogs, etc.
• Your impact ‘footprint’ is likely larger than you think, but it’s hard to
define
• Now is the time to start working on this if you haven’t already…
• Finally, if our work has influenced your practice, please get in touch!

41. THANK YOU
rob.farrow@open.ac.uk
All licences public domain / CC0 except where otherwise noted