Evaluation of the Grant Review Process. Richard Nakamura, Special Adviser, NIH Center for Science Review, USA

1. Evaluation of the Grant
Review Process
September 2019
Research on Research 2019
Richard Nakamura, Ph.D., Retired
Former Director, Center for Scientific Review, NIH

2. • Receives all NIH applications
• Reviews for scientific merit
about 75% of all NIH
applications
• >500 employees, $125M budget
• NIH spends more than $30B per
year on research
The Center for Scientific Review (CSR)
Gateway for All NIH Grant Applications

3. If the goal is to have grant funding
resources maximally advance
knowledge of living systems and
increase ability to understand and
treat disorders then finding the most
productive and efficient methods to
allocate resources to achieve those
goals is important. Experimental and
comparative methods are likely to be
most useful.

4. Improving Review: Goals, Measures, Tools
CSR developed multiple approaches to examine the
quality of review:
•Use of experts for quality measurement
•Use of quick feedback surveys
•Study efficiency of review
•Study ranking/scoring
•Assess fairness and reliability in peer review
•Evaluate of review alternatives (citations, AI)
•Reduction of burden on review process
4

5. Example: Fairness of Review

6. Fairness of Review
•In general, review is pretty fair
•Stage of career
•Gender
•Field
•Race/Ethnicity
•Reviewers

7. Probability of NIH R01 award by race and ethnicity, FY 2000 to FY 2006 (N = 83,188)
D K Ginther et al. Science 2011;333:1015-1019
Concerns of Reviewer Bias

8. The Hunt for Bias:
An Anonymizing Study

9. Anonymization Experiments –
Basic Assumptions
•Racial disparities in grant funding exists (Ginther et
al): AAs award rates much lower than Whites.
Other biases are suspected.
•Average preliminary overall impact scores account
for variance in final scores that account for award
disparity.
•The major hypotheses for score disparity were: a)
reviewer bias and/or, b) difference in quality of
application submission
•If there is bias, anonymizing applications should
reduce disparities.
9

10. Results and Lessons Learned

11. Need for Outcome Measures

12. Discussion

13. ORCID and interoperable vita systems
should be encouraged universally for
utility as long-term outcome
measures of scientific and technical
human capital.

14. Discussion

15. The Review Process at NIH

16. Spending
at NIH
NIH Extramural & Intramural Funding
FY 2017 Enacted: $33.4 Billion
83%
17%
Spending Outside NIH
$27.7 B
– Supports over 300,000 Scientists &
Research Personnel
– Supports over 2,500 Institutions
– $3.8 B Intramural Research
– $1.9 B Research Management & Support and
Other
$5.7 B

17. CSR Peer Review – Fiscal Year 2017
• 90,000 NIH applications received
• 61,000 applications reviewed by CSR
• 18,000 reviewers
• 247 Scientific Review Officers
• >200 standing and recurring study sections
• 1,600 review meetings

18. 2-Level Review System for NIH Grants
First Level of Review
Scientific Review Group
(Study Section) at CSR or IC
Second Level of Review
NIH Institute/Center Council
About 80,000 applications and 17,000 reviewers

19. Peer Review and Funding of NIH Grant Applications

20. The Study Section Meeting
The SRO Convenes the Study Section Meeting