This document discusses open science monitoring and the concept of research impact. It makes three key points: 1. Open science and impact are "boundary objects" that can be conceptualized differently across communities, so a one-size-fits-all approach should be avoided. 2. There are various ways to frame research impact, including dissemination, scientific effects, broader societal effects, extraordinary vs. mundane impacts, and impacts we can measure. 3. Monitoring open science includes tracking policies/initiatives, costs, uptake (through indicators like open access publications), and impacts both within and beyond academia (using bibliometrics, altmetrics, and other methods).

1. Open Science monitoring and
the notion of research impact
Marc Vanholsbeeck, Ph.D.
Director, Directorate of Scientific Research
Ministry of Wallonia-Brussels Federation,
Lecturer in Science Communication
Université Libre de Bruxelles

2. Open Science and research impact

3. ••• Open Science and Impact as boundary objects
• “Boundary Object” (Star and Griesemer, 1989)
– Common structure across “social worlds”
– Community-specific conceptualizations
• Hence better to avoid any “one size fits all”
approach in the implementation of OS policies
and the “impact agenda”

4. ••• Framing of impact (1)
• Impact as dissemination of research
• Impact as effects of research:
–scientific impact
–impact beyond academia:
• direct economic effects (income from
licenses, patents, spin-offs) (Benneworth,
2014)
• broader cultural, societal, health,
environmental and political effects

5. ••• Framing of impact (2)
• Impact as “extraordinary impact” vs.
manifold and mundane types of impact
(Sivertsen, 2017)
• Impact as the changes we can demonstrate,
measure and capture (Bayley, Phipps, Batac
& Stevens, 2018)

6. ••• Pathways to impact creation (1)
• Impact as « productive interaction » (Spaapen
& Van Drooge, 2011)
“Exchanges between researchers and stakeholders
in which knowledge is produced and valued that is
both scientifically robust and socially relevant;
Mediated through various ‘tracks’ (a research
publication, an exhibition, a design, people or
financial support);
Productive when it leads to efforts by stakeholders
to somehow use or apply research results or
practical information or experiences.”

7. ••• Pathways to impact creation (2)
• From direct instrumental impact to indirect
conceptual impact (King’s College London &
Digital Science, 2015)
• From linear (Caplan, 1977) to non-linear:
– Co-creation of impact with societal stakeholders
(Gronvad et al., 2017)
– Reacting to or driving societal changes (Muhonen
et al. 2010)

8. ••• Open Science in the creation of impact
beyond academia (1)
• Open access to knowledge ≠ discoverable
information
• Discoverable information ≠ efficient
communication
• Need of knowledge brokers and/or skills in
science communication

10. ••• Open Science in the creation of impact
beyond academia (3)

11. Monitoring of Open Science

12. ••• Open Science monitoring
• Monitoring of
– Open Science related policies and initiatives
– Costs implied by those policies and initiatives
– Uptake of the policies and initiatives
– Impact of Open Science on science and beyond
academia

13. ••• Monitoring OS related policies and
initiatives_ERA Roadmap 5th priority (1)
• Methodology
– ERA SWG Open Science and Innovation
– All Member States HR, LU, SK + CH, IS, MD, NO and TR
Has any activity for this
action taken place?
YES NO
Was the action delayed? YES NO YES NO
Is the action stopped? YES NO YES NO YES NO YES NO
Terminated
Results
On-
Going
with
Delay
Finished
Results
On
Track
Cancelled Postponed till Cancelled Scheduled for
(1) (2) (3) (4) (5) (6) (7) (8)

14. • Preliminary quantitative results
– 11 MSs completed the form
– 128 actions in total
– 53 actions concern exclusively OS (and not OI or KT)
– 33 actions involve OI (exclusively or in combination
with OS and/or KT)
– 53 actions on track
– 30 finished with results
– only 2 cancelled
••• Monitoring OS related policies and
initiatives_ERA Roadmap 5th priority (2)

15. ••• Indicators of Open Science uptake (1)
• Open Science Monitor

16. ••• Indicators of Open Science uptake (2)
• ERA Progress Report 2018
– Share of publications available in open access (headline
indicator)
– Share RFOs’ (i.e. members of Science Europe or other
important sources of national funding) publications that are
available in OA
– Share of life sciences papers to which a country contributed
and that have at least one open dataset in Figshare
• CWTS Leiden ranking 2019 (Van Leeuwen et al. 2019)
– Share of publications available in open access

17. ••• Monitoring Open Science impact (1)
• Bibliometrics and peer review for scientific
impact
• Economic metrics for economic impact
• Altmetrics to measure impact beyond academia?
(Miedema et al., 2018)

18. ••• Monitoring Open Science Impact (2)
Model of « World Class Research University »
Model of « Open University »
Scientific impact:
citations to articles in
international top
journals
Impact beyond
academia
Major publishers and
providers of « information
analytics »
Scopus
Web of
Science
Providers of altmetrics

19. “We need to start by asking: what science
communication model or paradigm can be
considered a suitable one for humanity?”
Arianna Becerril-García, Chair of AmeliCA (*)
*https://poynder.blogspot.com/2019/05/the
-oa-interviews-arianna-becerril.html

20. ••• Bibliography (1)
Bayley, J., Phipps, D., Batac, M., & Stevens, E. (2018). Development of a framework
for knowledge mobilisation and impact competencies. Evidence & Policy: A Journal
of Research, Debate and Practice.
Benneworth, P. (2014). Tracing how arts and humanities research translates,
circulates and consolidates in society. How have scholars been reacting to diverse
impact and public value agendas?. Arts and Humanities in Higher Education
1474022214533888, first published on May 14, 2014 as
doi:10.1177/1474022214533888.
Caplan, N. (1977). The use of social research knowledge at the national level. Social
Research in Public Policymaking,183 – 197.
Gronvad, J., Hvidtfeldt, R., & Pedersen, D.B. (2017). Analysing co-creation in theory
and in practice – A systemic review of the SSH impact literature. Retrieved
fromhttps://docs.wixstatic.com/ugd/35d470_78a9168f52d347de9ea1c8b2999882
6a.pdf?index=true
King’s College London, & Digital Science (2015). “The nature, scale and
beneficiaries of research impact An initial analysis of Research Excellence
Framework (REF) 2014 impact case studie”. Research Report 2015/01.
Miedema, F., Mayer, K., Holmberg, K., & Leonelli, S. (2018). Mutual Learning
Exercise: Open Science — Altmetrics and Rewards.
file:///C:/Users/vanhma01/Downloads/MLE%20OS_Final%20Report_0%20(4).pdf

21. ••• Bibliography (2)
Muhonen, R., Benneworth, P., & Olmos-Penuela, J. (2019). From productive
interactions to impact pathways: Understanding the key dimensions in
developing SSH research societal impact, Research Evaluation,
rvz003. https://doi.org/10.1093/reseval/rvz003
Sivertsen, G. (2017). Frameworks for understanding the societal relevance of the
humanities. Paper presented to RESSH2017 - Research Evaluation in the Social
Sciences and Humanities, Antwerpen, 6th7th July 2017.
Spaapen, J., & Van Drooge, L. (2011). Introducing ‘productive interactions’ in social
impact assessment. Research Evaluation, 20(3), 211-218.
SSH research societal impact.
Star, S. L., & Griesemer, J. R. (1989). Institutional ecology,translations' and
boundary objects: Amateurs and professionals in Berkeley's Museum of Vertebrate
Zoology, 1907-39. Social studies of science, 19(3), 387-420.
Tennant ,J.P., Crane, H., Crick, T., Davila J., Enkhbayar, A., Havemann, J., Kramer, B.,
Martin, R., Masuzzo, P., Nobes, A., Rice, C., López, B.R., Ross-Hellauer, T., Sattler, S.,
Thacker, P., Vanholsbeeck, M. (2019). Ten Hot Topics around Scholarly
Publishing. Publications, 7, 34. doi:10.3390/publications7020034
Van Leeuwen, T., Costas, R., & Robinson-Garcia N. (2019). Indicators of open access
publishing in the CWTS Leiden Ranking 2019. https://www.cwts.nl/blog?article=n-
r2w2a4.

22. •••