Ulo Palm, American Society for Quality (ASQ), ASQ Food, Drug & Cosmetic Division (FD&C) - Speaker at the marcus evans Evolution Summit 2012, held in Wheeling, IL, April 30 - May 2, 2012, delivered his presentation on Lack of Reproducibility in Biomedical Research – How a Common Quality Standard for Non-Regulated Biomedical Research Could Change the Trend

1. Lack Of Reproducibility In
p y
Biomedical Research –
How A Common Quality
Q y
Standard For Non-
Non-
regulated Biomedical
g
Research Could Change
The Trend
Ülo Palm, MD, PhD, MBA, CMQ/OE
Senior Member of the American Society for Quality
(ASQ)
Food, Drugs & Cosmetics Division (FD&C)
Marcus Evans
Senior Vice President Clinical Operations & Biometrics
p Evolution Summit,
Forest Research Institute April 30, 2012

2. Moore’s Law:
Transistor Count Doubling Every
Two Years
Source: Wikipedia

3. Eroom’s Law:
Number Of New Drug Approvals
Per Billion US Dollars Halved Every
Nine Years
Source: Nature Reviews – Drug Discovery: Diagnosing the decline in pharmaceutical R&D efficiency Vol 11, March
2012, page 191 - 200

4. “War On Cancer” Going On For 40 Years
War Cancer
• About $200 billion spent on cancer research in the
US since 1971*
US i 1971*
• More than 1.5 million papers published*
But despite some progress in prevention and
treatment
• Almost 1.5 million cancer cases and 560 000 cancer
deaths in the United States in 2009**
• Cancer now the second leading cause of death in
the US**
• Nearly 1 in 2 men and more than 1 in 3 women will
y
be diagnosed with cancer during their lifetime**
• Avastin increases survival in mCRC patients by 4.7
months*** at an annual treatment cost of
$58.000****
Source: * Fortune Magazine March 22, 2004, ** JAMA, March 17, 2010 - Vol 303, *** Avastin label, **** Reuters, Thu Jun 30, 2011

5. Some Causes Of Declining
Productivity Of Biomedical Research
• Biomedical R&D has become more
complex*
• Regulatory hurdles have gone up*
Regulatory hurdles have gone up
• Industrialization of drug discovery
(focus on single targets) has put
(focus on single targets) has put
R&D on an overall less effective
path
path*
• Pharmaceutical companies have
become too big to innovate
b t bi t i t
*Source: Nature Reviews – Drug Discovery: Diagnosing the decline in pharmaceutical R&D efficiency Vol 11, March
2012, page 191 - 200

6. Could there be another, major,
more fundamental root cause
of the declining productivity of
biomedical research?

7. Lack Of Reproducibility In
Biomedical Research
Bi di l R h
• “New Tools and technologies, massive amounts of
data, long‐term studies, interdisciplinary
d l di i di i li
approaches, and the complexity of questions being
asked are complicating replication efforts…”
• “An empirical assessment of 18 published papers of
“A ii l f 18 bli h d f
microarray studies showed that independent
analysts could perfectly reproduce the results of
only two of the studies…
only two of the studies ”
• “This is one of medicine's dirty
y
secrets: Most results, including
those that appear in top‐flight
peer‐reviewed journals, can't be
reproduced”

8. Lack Of Reproducibility:
Industry Experience
Amgen*
• Published literature described that inhibition of
theserine/threonine kinase 33 (STK33) destroyed cancer cells
• Amgen launched massive research effort but could not
replicate the results
p
Bayer*
• Bayer reported in September 2011 that it had halted nearly
two thirds of its early drug target projects because in house
two‐thirds of its early drug target projects because in‐house
experiments failed to match claims made in the literature.
Pfizer***
• Pfizer announced January 2012 that it had to write off $750
Pfizer announced January 2012 that it had to write off $750
million after results of a study with Dimebon for Alzheimer,
published originally in the journal Lancet*, could not be
reproduced
reproduced
Source: *WSJ, December 2, 2011; ** Fierce Biotech, January 17, 2012

9. Lack Of Reproducibility:
Case St d 1 f
C Study from A d
Academia i
• In 2002 paper published in Lancet by authors from
the FDA and NCI claimed that a mass spectrometry
method could provide highly sensitive and specific
diagnostic tests for ovarian cancer
g
• NCI announced a Clinical Proteomics initiative and
companies were formed to take assays based on this
method to the clinic
method to the clinic
• Independent analysis by MD Anderson researcher
demonstrated that the results were due to
experimental artifacts (running of all of the controls
before all of the cancers)
Source: K i h A . B
S Keith Baggerly and Kevin R . Coombes
l dK i C b
Handbook of Statistics in Clinical Oncology, Third Edition
Antje Crowley and John Hoering
Chapman and Hall/CRC 2012 Pages 605–618

10. Lack Of Reproducibility:
Case St d 2 f
C Study from A d
Academia i
• In 2006 a paper published in Nature Medicine by Duke
University professor Anil Potti claimed that microarray‐
based signatures of drug sensitivity derived from cell
lines could predict patient response to specific
chemotherapeutics
• Discover magazine designated this paper one of the top
100 breakthroughs of 2006
g
• Large clinical trials were started using this methodology
• In 2009, independent analysis demonstrated that the
data were wrong due to mislabeling and indexing errors
d t d t i l b li di d i
Source: Keith A . Baggerly and Kevin R . Coombes Baggerly KA, Coombes KR.
Handbook of Statistics in Clinical Oncology, Third Deriving chemosensiti it
Deri ing chemosensitivity from cell
Edition lines: Forensic bioinformatics and
Antje Crowley and John Hoering reproducible research in high-
Chapman and Hall/CRC 2012 Pages 605–618 throughput biology. Ann Appl Stat
2009; 3(4):1309–1334.

11. Non-reproducible Research Is
Noise And Not Knowledge
• What if the recent revelations indicate that most of
published biomedical research is noise?
– (“This is one of medicine's dirty secrets: Most
results, including those that appear in top‐flight
peer‐reviewed journals, can t be reproduced
peer‐reviewed journals can't be reproduced”
WSJ, December 2, 2011 )
• What is the noise/knowledge ratio of the 22 million
/ g
records in Medline? 30%?, 50%?, ….??
• How can biomedical research worldwide be
productive while trying to reproduce “noise”?
d ti hil t i t d “ i ”?
• What if the amount of “noise” in modern
g g y
biomedical research is beginning to suffocate any
productive research?

12. What are the reasons for the
lack f
l k of reproducibility in
d ibilit i
modern biomedical research?

13. Modern Science Is Based On
Collective I t lli
C ll ti Intelligence
• Collective intelligence requires:
– a shared body of knowledge, methods, and
y g
techniques, a shared praxis*
– a shared and agreed upon quality standard
how to plan, conduct, and report scientific
work
*Source: Reinventing Discovery – The new era of networked science by Michael Nielsen, 2012

14. Modern Biomedical Research
Does not Have A Well Defined
Quality Standard
• “Genes were mislabeled due to an off‐by‐one indexing
“G i l b l dd t ff b i d i
error”
• “We concluded that the method didn’t actually work at
all; it only appeared to work due to poor bookkeeping
all; it only appeared to work due to poor bookkeeping”
• “A disconnect between the numbers and the sample
names rendered the predictions invalid”
• “poor documentation allowed errors to go unnoticed
p g
until after things had proceeded to clinical trials “
• “the most common mistakes people make are simple
ones”
• “If the analyses are opaque, then the simple errors may
go unnoticed, and simple mistakes are still important”
Source: Keith A . Baggerly and Kevin R . Coombes, Handbook of Statistics in Clinical
Oncology, Third Edition, Antje Crowley and John Hoering
Chapman and Hall/CRC 2012 Pages 605–618

15. In 2000, The WHO Identified The Development Of A
Common Quality Standard For Biomedical Research As A
Q y
Pressing Global Need
• “The world’s population is facing serious
health challenges….. there is increased
demand for new drugs and new principles for
treatment …..it is essential that basic scientific
(biomedical) research as a whole, ……. be
(b d l) h h l b
conducted in a proper fashion using processes
that minimize waste of resources and reduce
the need for costly confirmation and
the need for costly confirmation and
repetition of work already performed”
• “It is hoped that wide application of the
“I i h d h id li i f h
quality practices proposed in this handbook
will lead to cost‐effective, accelerated
discovery research and will ultimately benefit
discovery research and will ultimately benefit
human health”
Source: Handbook: Quality Practices in Basic Biomedical Research (QPBR), WHO, 2006

16. The WHO Rationale For A
Quality St d d I Biomedical
Q lit Standard In Bi di l
Research
• To minimize waste of resources and
reduce the need for costly confirmation
and repetition of work already
d titi f k l d
performed
• To generate reliable data to ensure a
To generate reliable data to ensure a
solid basis for deciding whether to invest
in further development of a strategy or
in further development of a strategy or
product
• Quality means better science
Q y

17. The Flow Of Research Activities
From Planning To Publishing
Source: Handbook: Quality Practices in Basic Biomedical Research (QPBR), WHO, 2006

18. In 2009, a committee of the FD&C
Division of the American Society for
Quality (ASQ) was charged with
developing a Quality Standard for
Biomedical Research and Drug
Development in the US
“Best quality practices for biomedical research and
drug development”
1. ASQ Technical Report
2. ASQ Standard
3. ISO Standard
3 ISO St d d

19. “Best Quality Practices For Biomedical Research
And Drug Development
Development”
- Technical Report Content -
• Management system
g y • Document storageg
• Organization • Technical Requirements
• Project Management • Test Equipment
• Quality Management • Test methods / Method
System Validation
• Documentation • Sampling and Chain of Custody
Sampling and Chain of Custody
• Document control / • Materials
pp
Document approval and • Legal and Ethical
Legal and Ethical
issue: Considerations
• Document changes • Vendor Selection and
Qualification

20. The New Quality Standard for Biomedical
Research and Drug Development Will Be Based
esea c a d ug e e op e t e ased
On Current State-Of-The-Art Quality Practices
State-Of-The-
• World Health Organization Handbook: Quality Practices
World Health Organization Handbook: Quality Practices
for Biomedical Research
• ISO 17025
• BARQA Guidelines for Quality in Non‐Regulated Scientific
Research
• ICH Q2 Validation of Analytical Procedures: Text and
Methodology
• 21 CFR Part 58
• ICH Q9 Quality Risk Management
• ISO 900
ISO 900x

21. Co
Committee Members
ttee e be s
• George Bernstein, MAI
g , • Juli Motika, Regeneron
, g
Consulting • Ülo Palm, Forest Research
• Rick Calabrese, Sartorius‐ Institute
Stedim
St di • Michele Pruett, Innovative
• Keith Conerly, Sodexho Consultants GXP
• Li‐Chung Huang Eli Lilly
Li Chung Huang, Eli Lilly • Sandra R Storli Abbott Labs
Sandra R. Storli, Abbott Labs
• Alice Krumenaker, • John Surack, Clemson
CorePharma, LLC University
• Richard Lombardi, Forest • A. Mark Trotter, Trotter Biotech
Research Institute Solutions
• J
June Morita, Consultant
M it C lt t

22. Summary
• Declining productivity of Drug Development
Declining productivity of Drug Development
• Poor reproducibility of Biomedical Research
• Lack of a common language, a common quality
Lack of a common language, a common quality
standard in Biomedical Research
• Impacting medical progress and human health
• WHO developed Handbook on Quality Practices
in Basic Biomedical Research
• ASQ‐FD&C Division developing a new Quality
& d l l
Standard for Biomedical Research & Drug
Development in the US
Development in the US