Presentation at the OpenAIRE-COAR Conference: "Open Access Movement to Reality: Putting the Pieces Together", Athens - May 21-22, 2014. Session 4: The impact of openness and how to evaluate research. Next generation metrics of scholarly performance, by William Gunn - Head of Academic Outreach for Mendeley

1. Next-generation Metrics for
Scholarly Performance
William Gunn, Ph.D.
Head of Academic Outreach
Mendeley
@mrgunn
https://orcid.org/0000-0002-3555-2054

2. Open Access Policies

3. Open Access Has Benefits
• The perception that there were tradeoffs
when choosing open access slowed
adoption
• Only Open Access can fully provide the
benefits of the web – interoperability,
accessibility, discovery
• An expanded definition of impact brings
new kinds of content and new metrics

4. King, Christopher (2012) Thomson Reuters Annual Report
http://ar.thomsonreuters.com/_files/pdf/MultiauthorPapers_ChrisKing.pdf

5. Adams, Jonathan. "Collaborations: the fourth age of research."
Nature 497.7451 (2013): 557-560.

6.  Amgen: 47 of 53 “landmark” oncology publications could
not be reproduced
 Bayer: 43 of 67 oncology & cardiovascular projects were
based on contradictory results
 Dr. John Ioannidis: 432 publications purporting sex
differences in hypertension, multiple sclerosis, or lung
cancer. Only one data set was reproducible
There is no gold standard

7. Pochoda, Phil. "The big one: The epistemic system break in scholarly monograph
publishing." new media & society 15.3 (2013): 359-378.
Authors Publishers
Printers
Shippers
Booksellers
Readers

8. Pochoda, Phil. "The big one: The epistemic system break in scholarly monograph
publishing." new media & society 15.3 (2013): 359-378.
Authors Publishers
Printers
Shippers
Booksellers
Readers
1770

9. AuthorsReaders Internet
Research is changing
Publishers
Repositories
Professional
Websites

10. Challenges for Repositories
• The world is leaning green
• Repositories thus have an
essential role
• Repositories aren’t in the
workflow of a researcher, neither
in search for new ideas nor in
presentation of their work

11. The Ouroboros problem
• repositories need content
deposited to get usage, and
they need usage to get
deposits
• Discovery tools can address
usage and usage will
promote content
– scholarly outputs are papers,
datasets, code, and more.

12. New forms of discovery
• personalized recommendations based on
reading history
• related articles
• recommender frameworks
–implement recommendations as a
service
• third-party recommender services
–serve niche audiences

13. research impacts more than authors
http://altmetrics.org/manifesto/

14. ...and aggregates research
data in the cloud
Mendeley extracts
research data…
Install
Mendeley Desktop
Collecting rich signals
from domain experts.

15. Home Work Mobile
Cloud Library

16. Shared Folder

17. Mendeley Research Catalog

18. Mendeley Research Catalog

19. Read papers + keep track of notes
470M documents

20. Taking some misery out of writing

21. Information Extraction

22. We are publishing this data to the LOD cloud
http://code-research.eu/

23. Defining readership
• Each document addition is a
“read”
• stamped with metadata
describing the context of the read
event
• a read is like a citation, but faster
and captures more

24. research impacts more than authors
http://altmetrics.org/manifesto/

27. Readership vs. citations
• it comes with a payload of
metadata
• it accrues faster
• it illuminates previously
hidden impact

28. http://arxiv.org/html/1203.4745v1
altmetrics show broader impact

29. http://dx.doi.org/10.3789/isqv25no2.2013.04

30. http://www.asis.org/Bulletin/Apr-13/AprMay13_Lin_Fenner.html
altmetrics show broader impact

31. Issues To Be Addressed
• Identity
• Privacy
• Attribution
• Gaming
• Filtration
standards/
best practice

32.  Amgen: 47 of 53 “landmark” oncology publications could
not be reproduced
 Bayer: 43 of 67 oncology & cardiovascular projects were
based on contradictory results
 Dr. John Ioannidis: 432 publications purporting sex
differences in hypertension, multiple sclerosis, or lung
cancer. Only one data set was reproducible
There is no gold standard

33. Cultural skew is important
South America is weak on N.A. social media, strong on Mendeley

34. What would people build if they
could get the data?
• Impact Story – get credit for all your work
• PLOS ALM – article-level metrics for
papers
• Plum Analytics – bespoke analytics for
libraries (EBSCO)
• Altmetric.com – altmetrics for publishers.
(Digital Science)

35. www.mendeley.com
william.gunn@mendeley.com
@mrgunn