Presentation at the 23rd International Conference on Science and Technology Indicators (STI 2018). Leiden, the Netherlands, September 13, 2018.

1. Responsible metrics: One size doesn’t fit all
Ludo Waltman
Centre for Science and Technology Studies (CWTS), Leiden University
23rd International Conference on Science and Technology Indicators (STI 2018)
Leiden, the Netherlands
September 13, 2018

2. Responsible metrics
1

3. One size doesn’t fit all
• Research evaluation is an umbrella term
• Responsible use of scientometrics depends on evaluative setting
• Micro-level vs. macro-level evaluation
2

4. Macro-level research evaluation
• Comprehensive in-depth evaluation is
impossible at the macro level
• Experts view the world through
indicators
• Indicators do not just support
evaluation, they enable it
3Source: National Science Board
Science & Engineering Indicators 2018

5. Micro-level research evaluation
4
Whom to hire?

6. Micro-level vs. macro-level research evaluation
5

7. Implications of the micro-macro distinction
• Sophistication vs. simplicity
• Professional vs. citizen scientometrics
• Indicators vs. statistics
6

8. Sophistication vs. simplicity
• Macro level:
– Experts view the world through indicators
– Indicators need to have an unambiguous conceptual foundation and a high validity
– This typically requires a relatively high level of sophistication
• Micro level:
– Indicators do not so much provide information themselves; they point experts to potentially relevant
information
– Keeping indicators simple ensures that experts can truly reflect on what indicators tell them
7

9. Sophistication vs. simplicity
8

10. Professional vs. citizen scientometrics
• Citizen scientometrics not meant to be
understood in an offensive manner!
• Macro level:
– Need for professional scientometrics
• Micro level:
– To ensure indicators are helpful in supporting expert
assessment, involvement of experts in choice and design
of indicators is essential
– Experts need to take on role as citizen scientometrician
9

11. Indicators vs. statistics
• Terminology:
– Metric: Direct measurement
– Indicator: Proxy of an underlying concept
– Statistic: Quantitative summary of a body of information
• Macro level:
– Concepts of interest need to be clearly specified a priori
– Preferred term: Indicator
• Micro level:
– No need to impose strong a priori interpretations
– Interpretation is given by experts during evaluation
– Preferred term: Statistic
10

12. Conclusion
• Responsible use of scientometrics depends on evaluative setting
• Macro-level vs. micro-level distinction offers example
– Macro level requires ‘classical scientometrics’
– Micro level requires ‘contextualized scientometrics’
• Meso-level evaluation
• Interdependencies between levels
• Need for more refined taxonomies of different types of evaluations
11

13. Thank you for your attention!
12