Understand the foundations of bibliometrics, how these methods can be applied, and best practices for using these techniques.

1. —

2. •
•
•
•
•
AGENDA

3. — BACKGROUND
•
•

4. —
TRENDS IN SCHOLARLY RESEARCH

5. —
—
—
—
—
WHY DO INSTITUTIONS EVALUATE RESEARCH?

6. —
—
—
—
—
WHY DO INSTITUTIONS EVALUATE RESEARCH?

7. WHY USE BIBLIOMETRICS?

8. STAKEHOLDERS IN RESEARCH EVALUATION

9. WHAT DO CITATION COUNTS REALLY MEASURE?

10. WHAT DO CITATION COUNTS REALLY MEASURE?

11. —

12. BIBLIOMETRIC PERSPECTIVES ON SELF-CITATION
•
•
•
•
̶
̶
̶

13. —

14. USEFULNESS ≠ QUALITY

15. CITATION FREQUENCY CORRELATES WITH
OTHER MEASURES OF PEER ESTEEM

16. BEST PRACTICE: INFORMED PEER REVIEW

17. CITATION METRICS ARE ONE PIECE OF
THE RESEARCH PERFORMANCE PUZZLE.
—
—
—
•
•
•
•
•
—
•
•

18. — WEB OF SCIENCE
CITATION METRICS
1
•
•

19. CITATION METRICS ARE ONLY AS GOOD AS THEIR SOURCE

20. THE WEB OF SCIENCE CORE COLLECTION

21. SOURCE & FOUNDATION
̶
̶
̶

22. — APPROPRIATE USE
•
•

23. SAN FRANCISCO DECLARATION ON
RESEARCH ASSESSMENT (DORA)

24. —

25. THE LEIDEN MANIFESTO FOR RESEARCH METRICS

26. —
EXERCISES USING INCITES
•
•
•

27. USE A VARIETY OF INDICATORS

28. ADDITIONAL REFERENCE MATERIAL

29. — JOURNAL ANALYSIS
•
•

30. —

31. JOURNAL IMPACT FACTOR REFLECTS
A JOURNAL’S OVERALL PERFORMANCE

32. —
Eigenfactor score reflects a journal’s footprint in the overall journal-citation
network, measuring its influence in the entire network. It is based on the
Google PageRank method.
Journal C has a
higher PageRank or
“weight” than Journal
E, even though it has
fewer links citing it;
the one cite it does
have is of a much
higher value.
From Wikipedia
– “PageRank”
EIGENFACTOR SCORE

33. — NORMALIZED EIGENFACTOR SCORE
Normalized Eigenfactor
Score: a value of 1 indicates
average influence. A higher
value indicates above average
influence

34. — INSTITUTIONAL ANALYSIS
3
•

35. WHY NORMALIZE?
0
500
1000
1500
2000
0 3 6 9 12 15 18 21 24 27 30 33 36 39 42 45 48 51 54 57 60 63 66 69 72 75 78 81 84 87
0
500
1000
1500
2000
0 3 6 9 12 15 18 21 24 27 30 33 36 39 42 45 48 51 54 57 60 63 66 69 72 75 78 81 84 87
10
7.5

36. 36
ACCOUNT FOR FIELD, AGE, AND DOCUMENT TYPE

37. IS 20 CITATIONS GOOD OR BAD?
ACCOUNT FOR FIELD, AGE, AND DOCUMENT TYPE

38. Percentile in subject area
smaller is better — in this example 0.04% of the papers in the
category (plant sciences) in 2014 had more citations
ACCOUNT FOR FIELD, AGE, AND DOCUMENT TYPE
Average percentile
for a group of papers, we average all of the documents’
percentiles

39. — AUTHOR ANALYSIS
3
•

40. AUTHOR IDENTIFICATION
40
Author clustering
-applied to Web of Science
Core Collection regularly
6.2 million Web
of Science Core
Collection records
•722,000
ResearcherIDs
•7.7 million Web
of Science Core
Collection records
AUTOMATED AUTHOR VERIFIED

41. THE H-INDEX

42. DETERMINING H-INDEX

43. —
—
•
•
—
•
—
USING H-INDEX IN RESEARCH EVALUATION

44. THESE THREE AUTHORS HAVE THE SAME H-INDEX (4)

45. THANK YOU

46. — APPENDIX
4

47. —

48. WHY CITATION METRICS?
—
—
—

49. —

50. Bornmann, L., & Marx, W. (2015). Methods for the generation of normalized citation impact scores in bibliometrics: Which method best reflects the judgements of experts? Journal of
Informetrics, 9(2), 408-418. doi:10.1016/j.joi.2015.01.006
Clark, K.E. (1957). The APA study of psychologists. American Psychologist, 9, 117–120.
Cole, S., and Cole, J.R. (1967). Scientific output and recognition: A study in the operation of the reward system in science. American Sociological Review, 32, 377–390.
Derrick, G. E., Haynes, A., Chapman, S., & Hall, W. D. (2011). The Association between Four Citation Metrics and Peer Rankings of Research Influence of Australian Researchers in Six
Fields of Public Health. Plos One, 6(4). doi:10.1371/journal.pone.0018521
Garfield, E., and Welljams-Dorof, A. (1992a). Of Nobel class: A citation perspective on high impact research authors. Theoretical Medicine, 13, 118–126.
Lovegrove, B. G., & Johnson, S. D. (2008). Assessment of research performance in biology: How well do peer review and bibliometry correlate? Bioscience, 58(2), 160-164.
doi:10.1641/b580210
Mryglod, O., Kenna, R., Holovatch, Y., & Berche, B. (2013). Absolute and specific measures of research group excellence. Scientometrics, 95(1), 115-127. doi:10.1007/s11192-012-
0874-7
Norris, M., & Oppenheim, C. (2010). Peer review and the h-index: Two studies. Journal of Informetrics, 4(3), 221-232. doi:10.1016/j.joi.2009.11.001
Oppenheim, C. (1997). The correlation between citation counts and the 1992 research assessment exercise ratings for British research in genetics, anatomy and archaeology. Journal of
Documentation, 53(5), 477-487. doi:10.1108/eum0000000007207
Small, H.G. (1977). Co-citation model of a scientific specialty: – a longitudinal study of collagen research. Social Studies of Science, 7 (2), 139–166.
Smith, A.T., and Eysenck, M. (2002). The correlation between RAE rankings and citation counts in psychology. Technical Report, Psychology, University of London, Royal Holloway.
http://cogprints.ecs.soton.ac.uk/archive/00002749/01/citations.pdf
Van Raan, A. F. J. (2006). Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups. Scientometrics, 67(3), 491-
502. doi:10.1556/Scient.67.2006.3.10
Vieira, E. S., Cabral, J. A. S., & Gomes, J. (2014). How good is a model based on bibliometric indicators in predicting the final decisions made by peers? Journal of Informetrics, 8(2),
390-405. doi:10.1016/j.joi.2014.01.012
Virgo, J. A. (1977). A Statistical Procedure for Evaluating the Importance of Scientific Papers. Library Quarterly, 47 (4), 415-430.
VALIDATION STUDY BIBLIOGRAPHY

51. THOMSON REUTERS – THE AUTHORITY ON CITATION DATA

52. KEEP DATA COLLECTION AND ANALYTICAL PROCESSES
OPEN, TRANSPARENT, AND SIMPLE

53. ACCOUNT FOR VARIATION BY FIELD IN PUBLICATION AND
CITATION PRACTICES
Understand the subject areas you are using
InCites subject area schemes Source Type
Web of Science Thomson Reuters Journal to category
Essential Science Indicators Thomson Reuters Journal to category
Global Institutional Profiles Project (GIPP) Thomson Reuters Category to category
ANVUR Italy Category to category
Australia ERA Australia Journal to category
China SCACD China Category to category
FAPESP Brazil Category to category
OECD (Frascati) OECD Category to category
UK RAE and REF UK Category to category
KAKEN Japan Category to category

54. BIBLIOMETRIC PERSPECTIVES ON “NEGATIVE” CITATION

55. citations
citation stacking
article-type manipulation
self-citation false publications
mentions
Purchased likes mention bots
Purchased
retweets
social media promotion tools
& Bots
Usage
Bots
Harvesters & scrapers