How do Scholars Evaluate and Promote Research Outputs? An NTU Case Study Authors: Han Zheng, Mojisola Erdt, Yin-Leng Theng Workshop Website: http://www.altmetrics.ntuchess.com/AROSIM2018/

1. How do Scholars
Evaluate and Promote
Research Outputs?
An NTU Case Study
Han Zheng, Mojisola Erdt, Yin-Leng Theng
Wee Kim Wee School of Communication and
Information
Nanyang Technological University, Singapore
AROSIM, January 2018

2. 2
Outline
1. Introduction
2. Objectives and Research Questions
3. Methodology
4. Findings
5. Conclusion

3. 3
1. Introduction
Research Outputs Dissemination
 Traditionally, scholars use several methods to promote their research outputs
such as attending academic conferences, publishing papers in academic
journals, and networking with other researchers in research seminars.
 With the coming of social media to facilitate communication, scholars have
also realised the benefits of such media tools and have embarked on using
them in their academic lives.
 The dissemination of research outputs is becoming more effective and
convenient, regardless of geographical or temporal distance.

4. 4
1. Introduction
Research Outputs Evaluation
 In the past, research outputs were evaluated by a variety of bibliometrics
including citation counts, Journal Impact Factor (JIF), H-index, etc. These
metrics have been used by scholars across different academic disciplines for
decades and could be said to be significant indicators of the impact of
research works.
 In recent years, owing to the popularity of social media among scholars,
altmetrics, indices based on social media, have been emerging in academia as
a new approach to measuring research outputs.

5. 5
2. Objectives and Research Questions
With the aim to understand whether the advent of social media has changed
the way of promoting and evaluating research outputs in academia, this study
adopted a qualitative approach to investigating how researchers at Nanyang
Technological University (NTU), Singapore, promote research outputs and
how they conduct research evaluation.
Research Questions
RQ1: How do NTU researchers promote their research outputs?
RQ2: How do NTU researchers evaluate research outputs?

6. 6
3. Method
Study Design
Approach: Semi-structured interviews (10 - 20 minutes)
Target Groups: Faculty and Non-faculty staff employed at NTU
NTU IRB (Institutional Review Board): Approved in February 2017
Data Collection Period: March – June 2017 (4 months) for this study
Faculty
Professors
Associate
Professors
Assistant
Professors
Lecturers
Instructors
Non-faculty
Research
Staff
Librarians
Research
Students
Others

7. 7
4. Findings
Interview Participants
Total % Participants
Gender
Female
Male
5
13
28
72
P1, P11, P15, P16, P18
P2, P3, P4, P5, P6, P7, P8, P9,
P10, P12, P13, P14, P17
Discipline
Hard Sciences (sciences, computing
and engineering)
Non-hard Sciences (humanities, arts,
business and social sciences)
9
9
50
50
P1 – P9
P10 – P18
Career Level
Faculty
Non-faculty
6
12
33
67
P7, P8, P9, P16, P17, P18
P1 – P6 & P10 – P15
Demographics of the interview participants (N = 18)

8. 8
4. Findings
RQ1: How do NTU researchers promote their research outputs?

9. 9
4. Findings
RQ2: How do NTU researchers evaluate research outputs?
 Traditional metrics such as citation counts and JIF are accepted and used for
research evaluation by all participants from NTU.
 Most participants (16 out of 18) believed that traditional metrics are good
indicators to determine the quality of the research paper.
 Limitations of traditional metrics such as the easiness of getting higher citation
counts in certain disciplines and the reliability of these metrics were also
highlighted.

10. 10
4. Findings
RQ2: How do NTU researchers evaluate research outputs?
 Participants have not yet used altmetrics as formal indicators for research
evaluation. However, one third of them (6) mentioned that altmetrics could be
a good tool to complement traditional metrics.
 A few interviewees from non-hard sciences indicated that they were willing to
use altmetrics in the future if NTU were to establish relevant policies on the
adoption of altmetrics in the research evaluation process.
 Reasons for altmetrics not being accepted in academia:
1. Strong reliance on traditional metrics;
2. The quality of altmetrics.

11. 11
5. Conclusion
Key Findings
 We have conducted semi-structured interviews with NTU researchers to
understand how they promote and evaluate research outputs.
 Attending academic conferences and publishing paper in journals are the two
main methods to promote research findings among NTU researchers.
 Academic social networking sites like ResearchGate and Google Scholar are
used as an alternative way to promote their research.
 Traditional metrics such as citation counts and JIF are frequently used by most
NTU researchers in their professional lives.
 Researchers at NTU have been aware of the existence of altmetrics on social
media; however, they have not considered using them as an alternative
method to evaluate research currently.

12. 12
5. Conclusion
Limitations
 The sample size of this study is too small (N = 18).
 The case study was conducted at only one university (NTU).
Future Directions
 Future study should include more scholars from different disciplines and
institutions.
 A survey study can be conducted to understand the awareness and usage of
traditional metrics and altmetrics among scholars.

13. 13
Acknowledgements
 This research is supported by
the National Research
Foundation, Prime Minister’s
Office, Singapore under its
Science of Research,
Innovation and Enterprise
programme (SRIE Award No.
NRF2014-NRF-SRIE001-019).
 Project Duration: Feb 2015 to
Mar 2018
Downtown Singapore
Nanyang Technological University
Altmetrics: Rethinking and Exploring New Ways of
Measuring Research

14. 14
Contact Information
Contact Details
 Prof. Theng Yin Leng
Principal Investigator
Director, Centre for HEalthy & Sustainable CitieS (CHESS)
Wee Kim Wee School of Communication & Information
Nanyang Technological University, Singapore
Email: tyltheng@ntu.edu.sg
Tel: (65) 6790-5834
 Dr. Helen Mojisola Erdt
Research Fellow
Centre for HEalthy & Sustainable CitieS (CHESS)
Wee Kim Wee School of Communication & Information
Nanyang Technological University, Singapore
Email: Mojisola.Erdt@ntu.edu.sg
 Han Zheng
Research Assistant
Centre for HEalthy & Sustainable CitieS (CHESS)
Wee Kim Wee School of Communication & Information
Nanyang Technological University, Singapore
Email: zhenghan@ntu.edu.sg
More Info:
 CHESS
 http://www.wkwsci.ntu.edu.sg/Research/Pages/Resear
ch-Centres.aspx
 http://www.chess.wkwsci.ntu.edu.sg
 Altmetrics Project
 http://www.wkwsci.ntu.edu.sg/Research/Pages/Altmet
rics-Rethinking-and-Exploring-New-Ways-of-Measuring-
Research.aspx
 http://www.altmetrics.ntuchess.com

15. 15
THANK YOU