There has been a significant decline of productivity in pharmaceutical R&D • Many different reasons have been discussed, but little attention has been paid to the role of poor reproducibility of biomedical research in general • There are often fundamental doubts about the credibility of research data leading to costly and often futile repetition of biomedical research studies • One of the root causes for the reproducibility problem is the lack of a common quality standard for biomedical research • There have been recent initiatives by the WHO and the FD&C Division of the American Society for Quality (ASQ) to address this issue • The pharmaceutical industry needs to embrace these initiatives in order to reverse the negative productivity trend

1. The Reproducibility Crises
in Biomedical Research
and its Impact on
Pharmaceutical R&D
Productivity
Ülo Palm, MD, PhD,
Senior Vice President Clinical Operations & Biometrics
Forest Research Institute
February 11, 2013
1

2. Moore’s Law:
Transistor Count Doubling Every Two
Years
Source: Wikipedia 2

3. Eroom’s Law:
Number Of New Drug Approvals
Per Billion US Dollars Halved
Every Nine Years
3
Source: Nature Reviews – Drug Discovery: Diagnosing the decline in pharmaceutical R&D efficiency Vol 11, March
2012, page 191 - 200

4. “ War On Cancer” Going On For 40 Years
• About $200 billion spent on cancer research
in the US since 1971*
• More than 1.5 million papers published*
But despite some progress in prevention and
treatment
• Nearly 1 in 2 men and more than 1 in 3
women diagnosed with cancer during their
lifetime**
• Almost 1.5 million cancer cases and 560 000
cancer deaths in the United States in 2009**
• Cancer now the second leading cause of
death in the US**
4
Source: * Fortune Magazine March 22, 2004, ** JAMA, March 17, 2010 - Vol 303,

5. Some Causes Of Declining
Productivity Of Biomedical
Research
• Biomedical R&D increasingly
complex*
• Higher regulatory hurdles *
• Pharmaceutical companies too big
to innovate
5
*Source: Nature Reviews – Drug Discovery: Diagnosing the decline in pharmaceutical R&D efficiency Vol 11, March
2012, page 191 - 200

6. Is There Another Major, Root Cause Of The
Declining Productivity Of Biomedical Research?
6

7. Lack Of Reproducibility In Biomedical
Research – Literature And Press:
“Five of the seven largest published studies
addressing cancer prognosis did not classify
patients better than chance”
7

8. Lack Of Reproducibility In Biomedical
Research – Literature And Press:
8

9. Lack Of Reproducibility In Biomedical
Research – Literature And Press:
November 2010
9

10. Lack Of Reproducibility In Biomedical
Research – Literature And Press:
December 2011
“This is one of medicine's dirty secrets: Most results,
including those that appear in top-flight peer-reviewed
journals, can't be reproduced”
10

11. Lack Of Reproducibility In Biomedical
Research – Literature And Press:
December 2011
“An empirical
assessment of 18
published papers
of microarray
studies showed
that independent
analysts could
perfectly
reproduce the
results of only two
of the studies…”
11

12. Lack Of Reproducibility In Biomedical
Research – Literature And Press:
2012
“Forensic bioinformatics is the art of using
raw data and reported results to infer what
must have happened to get from one to the
other”
12

13. 13

14. Lancet 2002
• A new mass spectrometry method a highly
sensitive and specific diagnostic tests for ovarian
cancer
• NCI launched clinical Proteomics initiative
• Companies formed to take assays based on this
method to the clinic
• Turned out to be an experimental artifact due to
running of all of the controls before all of the
cancers
Source:
Keith A . Baggerly and Kevin R . Coombes
Petricoin EF, Ardekani AM, Hitt BA, Levine PJ,
Handbook of Statistics in Clinical Oncology, Third
Fusaro VA, Steinberg SM, Mills GB et al. Use of
Edition
proteomic patterns in serum to identify ovarian
Antje Crowley and John Hoering 14
cancer. Lancet 2002; 359(9306):572–577
Chapman and Hall/CRC 2012 Pages 605–618

15. Nature Medicine 2006
• Microarray-based signatures of drug sensitivity
derived from cell lines predict patient response to
specific chemotherapeutics
• Discover magazine designated this paper as one
of the top 100 breakthroughs of 2006
• Large clinical trials started using this
methodology
• In 2009, independent analysis demonstrated that
the data were wrong due to mislabeling and
indexing errors
Source:
Keith A . Baggerly and Kevin R . Coombes Potti A, Dressman HK, Bild A, Riedel RF, Chan
Handbook of Statistics in Clinical Oncology, Third G, Sayer R, Cragun J et al. Genomic signatures
Edition to guide the use of chemotherapeutics. Nat Med
Antje Crowley and John Hoering 2006; 12:1294–1300. Retracted January 7,
Chapman and Hall/CRC 2012 Pages 605–618 2011. 15

16. A few “What If Questions”
16

17. Impact On Industry
Productivity
Amgen*
• Published literature described that inhibition of STK 33
destroyed cancer cells
• Amgen could not replicate the results despite massive
research efforts
Bayer*
• Bayer halted in 2011 nearly two-thirds of its early drug target
projects because in-house experiments failed to match claims
made in the literature.
Pfizer***
• Pfizer wrote off $750 million in 2012 after results of a study
with Dimebon for Alzheimer, published originally in the
journal Lancet*, could not be reproduced
Source: *WSJ, December 2, 2011; ** Fierce Biotech, January 17, 2012
17

18. What Are The Reasons For
The Lack Of Reproducibility
In Modern Biomedical
Research?
18

19. Modern Science Is Based On
Collective Intelligence
• Collective intelligence requires:
– a shared body of knowledge, methods,
and techniques, a shared praxis*
– a shared and agreed upon quality standard
how to plan, conduct, and report scientific
work
*Source: Reinventing Discovery – The new era of networked science by Michael Nielsen, 19
2012

20. No Well Defined Quality Standard
Exists In Biomedical Research
• “Genes were mislabeled due to an off-by-one
indexing error”
• “The method didn’t actually work at all; it only
appeared to work due to poor bookkeeping”
• “A disconnect between the numbers and the
sample names rendered the predictions
invalid”
• “Poor documentation allowed errors to go
unnoticed until after things had proceeded to
clinical trials “
Source: Keith A . Baggerly and Kevin R . Coombes, Handbook of Statistics in Clinical
Oncology, Third Edition, Antje Crowley and John Hoering
Chapman and Hall/CRC 2012 Pages 605–618 20

21. WHO: Development Of A Common Quality Standard
For Biomedical Research A Pressing Global Need
• “The world’s population is facing
serious health challenges…
• .. there is increased demand for new
drugs and new principles for
treatment …
• ..it is essential that basic scientific
(biomedical) research , …is conducted
in a proper fashion… using processes
that minimize waste of resources
• ….to reduce the need for costly
repetition of work already
performed”
Source: Handbook: Quality Practices in Basic Biomedical Research (QPBR), WHO, 2006 21

22. American Society For Quality (ASQ) : Best Quality
Practices For Biomedical Research In Drug
Development
• The objective is to increase the
overall productivity of biomedical
research by promoting a common
quality standard:
• “A common quality guideline for
biomedical research in drug
development will ensure the
validity and credibility of
scientific data from different
research institutions and
facilitate the mutual acceptance
Source: http://asq.org/quality-
press/display-item/index.html? of research results”
item=T919E
22

23. A Vision For The Future
Standardized and
validated internal
research
End-to-end electronic
•Integration
Electronic
•Analysis
Submission
•Modeling
•Simulation
Standardized and
validated external
research
23

24. Summary Of Issues
• Declining productivity of Drug
Development
• Poor reproducibility of Biomedical
Research
• Lack of a common quality standard in
Biomedical Research
24

25. Solution
Establish a common quality standard for
biomedical research in drug development
•WHO and ASQ published quality guidelines for
Biomedical Research
•Pharmaceutical Industry should promote these
guidelines to improve R&D productivity
25

26. Thank You!
26

27. Backup slides
27

28. The Flow Of Research Activities
From Planning To Publishing
Source: Handbook: Quality Practices in Basic Biomedical Research (QPBR), WHO, 2006

29. “ Best Quality Practices For Biomedical Research
And Drug Development”
- Technical Report Content -
• Management system • Document storage
• Organization • Technical Requirements
• Project Management • Test Equipment
• Quality Management • Test methods / Method
System Validation
• Documentation • Sampling and Chain of Custody
• Document control / • Materials
Document approval and • Legal and Ethical
issue: Considerations
• Document changes • Vendor Selection and
Qualification