Subtitle: What metrics based on publication and citation data can and can't tell us. Following on from DORA (2013), the idea of responsible metrics has developed a clear voice in publications such as the Leiden Manifesto (2015) and the Metric Tide (2015). This breakout session described how library staff at Lancaster University became involved in a seminar series entitled ‘Designing the academic self’ (2016), and contributed insights into what metrics based on publication and citation data can and can’t tell us. Participants took part in the discussion-based activities that were used to illustrate the uses and limitations of quantitative indicators in the original seminar. Presented as a breakout session at Bibliometrics in Practice event, Manchester Metropolitan University, UK on 27th June 2016. Organised by LIS-Bibliometrics Committee.

1. ‘Designing the
academic self’
What metrics based on
publication and citation
data can and can’t tell us
Tanya Williamson
Academic Liaison Librarian: Research
Lancaster University Library
Bibliometrics in Practice, 27th June 2016

2. I will describe how library staff at Lancaster University
became involved in a seminar series entitled ‘Designing the
academic self’, and contributed insights into what metrics
based on publication and citation data can and can’t tell us.
Participants will take part in the discussion-based activities
that were used to illustrate the uses and limitations of
quantitative indicators in the original seminar.

3. • Context: Designing the Academic Self seminar series
• What questions can metrics help us to answer?
– With a discussion activity in pairs/small groups
• What questions can’t metrics help us to answer?
– With a discussion activity in two groups
• Summary and discussion

4. Designing the Academic Self
Idea for a seminar series developed from ESRC-funded research
‘The Dynamics of Knowledge Creation: Academics' Writing
Practices in the Contemporary University Workplace’ in the
Department of Linguistics
Session 1: Who does the Internet think you are?
Session 2: How are metrics affecting academics’ writing
practices in UK universities?
Session 3: What can and can’t metrics tell us?
Session 4: Metrics through a critical lens
Project blog https://wp.lancs.ac.uk/acadswriting/

5. Citation analysis
Citation analysis is the quantitative analysis of citations to
publications.
Based on the assumptions that:
publications = productivity
citations = positive academic impact

6. What questions can metrics help
us to answer?
What do you think?
• How many…?
• How often…?
• What’s the average…?
• How many … compared to …?
• What are the trends?
According to the data source

7. Activity
Choose a role.
Discuss with a partner:
‘What questions might
you wish to answer
using this kind of
data?’
5 mins

8. What can’t metrics help us to
answer?
What do you think?
• Why was this work cited?
• What did they think?
• What was the influence, i.e. how did it change thought or
practice?
• Why are the metrics low/high?
• What will the long term, or delayed impact be?
• What is the societal impact?
If it’s not included in the data source, it is not counted

9. Activity
• Group 1: Why might
a particular research
output be highly
cited?
• Group 2: Why might
a particular research
output receive few or
no citations?
5 mins
0
5
10
15
20
25
30
35
1990
1995
2000
2005
2010
2015
Output A B C

10. The assumption that citations = positive academic
impact is usually, but not always true
Why else might an item be cited?

11. Retraction Watch
http://retractionwatch.c
om/the-retraction-
watch-leaderboard/top-
10-most-highly-cited-
retracted-papers/

12. Does publication always = productivity?
What kind of ‘productive’ activities might be missed?
Is publication of co-authored journal articles the norm
in your discipline?
What if your outputs aren’t recorded in the data sources?

13. http://imagination.lancs.ac.uk/news/Leapfrog_Young_Designers_Celebration_Event
http://impact.lancaster.ac.uk/tools/#/agenda
http://www.lancaster.ac.uk/users/ruskinlib/Pages/power.html
http://www.transculturalwriting.com/radiophoni
cs/contents/onlineworkshops/index.html
Productive work you won’t
find in Scopus

14. Summary
• Librarians can have a role in challenging assumptions around
bibliometrics
• It’s possible to be critical without being negative!
• Simple activities can help us get across concepts around the
responsible use of metrics
Share, adapt, remix with attribution, non-commercial