The document discusses several common pitfalls in preclinical cancer research studies that can lead to inaccurate or irreproducible results. These include:
1) Failing to establish a causal rather than just correlative relationship between a target and cancer phenotype.
2) Lack of robustness, as results are not tested under diverse conditions.
3) Inability to distinguish on-target from off-target effects, particularly for assays measuring decreases in signals.
4) Overreliance on statistical significance without consideration of biological or clinical significance.
2. ď§ I am not a cancer biologist, Iâm a âbiology watcherâ.
ď§ Some of you are card-carrying cancer biologists.
ď§ Corollary: You know more cancer biology than I do.
ď§ So feel free to interject, educate, enlighten.
2
3. ď§ Efficacy still leading cause of failure in drug discovery.
ď§ Reproducibility of preclinical cancer studies depressingly low:
ď§ Amgen study: Only 6 out of 53 âlandmarkâ cancer studies
could be replicated.
ď§ Bayer study: Only 25% of preclinical studies had data that
warranted further continuation of projects.
ď§ Bad preclinical data = bad clinical trials. Huge potential
negative impact on patients.
3
4. ď§ Robustness: Ability of a system (eg. aircraft, Internet) to
withstand perturbations, removal of key components.
ď§ Robustness in cancer studies: Data must be gathered
under diverse conditions.
ď§ Diverse cell lines
ď§ Diverse assay and buffer conditions
ď§ Diverse tissue and patient samples.
ď§ Important for scientists to test for and report robustness â
and lack thereof.
ď§ Unfortunately, journals actively discourage publication of
âconflictingâ data that may not only indicate degree of
robustness but also point to unexpected and novel
findings.
4
7. 7
âCommon pitfalls in preclinical cancer target validationâ
Kaelin, W. G. Nat. Rev. Cancer, 2017, 17, 441
8. ď§ Can emerge simply by chance. Eg. Identification of patient
subpopulations for âprecision medicineâ: Retrospective
evidence not enough, tests need to be re-run for statistical
significance.
ď§ Hypoxia-Inducible Factor (HIF) and hypoxia. Possibilities:
ď§ HIF upregulation and hypoxia cause tumor progression.
ď§ Tumor progression causes HIF upregulation and hypoxia (reverse
causation).
ď§ Only hypoxia promotes tumor growth. HIF merely biomarker.
ď§ Reality: HIF correlates with proteins causing tumor growth and
regression!
ď§ Distinguishing cause, consequence and irrelevant correlations
key and non-trivial (eg. Ă amyloid and Alzheimerâs).
8
9. ď§ Phrases like âlinked toâ, âassociated withâ
and âconsistent withâ maddeningly
confusing. Important to know whether A
necessary or sufficient (or both) for B.
ď§ Eg. BRAF mutations found in both
malignant and benign melanoma. Thus:
ď§ BRAF not sufficient for melanoma maintenance.
ď§ BRAF necessary for melanoma maintenance.
ď§ Distinguishing necessary from sufficient
important in connecting multiple driver
mutations to multidrug therapy.
9
âCommon pitfalls in preclinical cancer target validationâ
Kaelin, W. G. Nat. Rev. Cancer, 2017, 17, 441
10. ď§ Assays can look for either increase or decrease of particular
signals (eg. protein production, fluorescence, apoptosis).
ď§ Signal in down assays can be swamped by noise.
ď§ More important: Down assays likely to suffer from multiple off-
target causes, can produce results with trivial/uninteresting
explanations.
ď§ Reality: Many assays used by cancer biologists are down assays,
measuring decreases in protein phosphorylation, cell viability
etc.
ď§ Important to know whether results of down assays are real and
on-target/biologically interesting.
10
âCommon pitfalls in preclinical cancer target validationâ
Kaelin, W. G. Nat. Rev. Cancer, 2017, 17, 441
12. ď§ Another graveyard for distinguishing correlation and causation
between target modulation and phenotype.
ď§ Are your down assays producing results because of off-target
effects? Especially true at higher drug concentrations.
ď§ Potential experiments to validate on-target effects:
ď§ Rescue experiments: eg. BCR-ABL variants resistant to imatinib make
CML cells resistant.
ď§ Restoring downstream effector function: Eg. Activation of MEK1 makes
mutant melanoma cells resistant to BRAF inhibitors.
ď§ Establishing correlation between gain of function and opposite
phenotype (Caveat: Overexpressing some targets might cause loss of
general cell function because of overload).
12
14. ď§ Positive controls can distinguish between technical assay
failures and real effects of perturbant.
ď§ Negative controls can test for specificity: eg. distinguish
between various perturbants.
ď§ Negative controls particularly valuable in down assays,
especially when tested as minor variations (eg. as
enantiomers).
ď§ When possible, different kinds of controls (genetic, chemical,
antibodies etc.) should be tested for corroborative evidence.
ď§ Whatâs the null model?:
ď§ Probability you got the result by chance?
ď§ Did you use the right vehicle?
14
15. ď§ Role of target in development not always predictive of role in
adult: eg. Germline knockout of Abl embryonic lethal in mice,
imatinib modulation viable in humans.
ď§ Therapeutic vs toxic response all about right level of target
engagement: Eg. Proteasome inhibitor Bortezomib viable at
concentration that reduces 50-80% of proteasome activity, toxic
at higher concentration.
ď§ Animal models not always predictive of humans: Eg. Development
of imatinib delayed due to liver toxicity seen only in animal
model.
ď§ Ultimately, therapeutic window dictated not by target
requirement in normal cells but by difference between
requirements in normal vs cancer cells.
15
16. 16
âThere are known knowns; there are things we know we
know. We also know there are known unknowns; that is to
say we know there are some things we donât know. But there
are also unknown unknowns â the ones that we donât know
we donât know.â
- Donald âIraq Warâ Rumsfeld
18. ď§ Known knowns: Targets whose necessary causal connections to
disease states have been established through a variety of means.
BCR-ABL, HER2, EGFR, PD-1?
ď§ Note: Truly speaking, known known targets validated only years after
drug hits market, and perhaps neverâŚ
ď§ Known unknowns: Targets whose general role is known but whose
effects on particular disease states are unknown, especially when
perturbing other related targets.
ď§ Unknown unknowns: Targets whose existence per se may or may not
be known, but whose role in disease states may not even be
suspected.
ď§ Unknown knowns: Targets whose role in disease states you think
you know, which may be doing something completely different.
18
19. ď§ System rewards positive results, actively discourages negative results.
ď§ âTechnological solutionismâ: Easy to do experiments, therefore do
experiments without thinking whether theyâre asking the right
questions (eg. CRISPR, chemical genetics, HTS).
ď§ âDataismâ: Use data because itâs available, not necessarily because it
means something relevant (eg. âtargetsâ from cheap genomic
sequencing).
ď§ Ignoring statistics: eg. excessive reliance on p values, failure to
distinguish between statistical and real life significance etc.
âThe first question that should be asked when assessing the importance of a
paper is whether its findings are likely to withstand the test of time, not
which journal it appeared inâŚwe should lower the bar with respect to the
number of claims required for publication and raise the bar in terms of the
burden of proof required to support those claimsâ â William Kaelin
19
Morozov, E. âTo Save Everything, Click Here: The Fallacy of Technological Solutionismâ (2013)
Yaffe, M. âThe Scientific Drunk and the Lamppostâ Sci. Signal. 2013, 2, 269
Zilliak, S. and McCloskey, D. âThe Cult of Statistical Significanceâ (2008)
20. ď§ Is our target merely correlated with the phenotype or causing it?
ď§ Is the target-phenotype relationship robust, irrespective of effect size? Does it hold up
under multiple conditions, cell types, assays, animal models etc.?
ď§ Is there a clear causal relationship between target and phenotype in our down
assays?
ď§ Are we clearly distinguishing between on-target and off-target effects?
ď§ Can we use up assays instead?
ď§ Is our target necessary, sufficient or both for effecting the phenotype?
ď§ Are we testing enough positive and negative controls? Both genetic and chemical?
ď§ Do our assays/studies have good statistical validation?
ď§ Occamâs Razor: Is there a simpler explanation for the results?
ď§ Psychology: Are we falling for conventional wisdom? Groupthink? Confirmation bias?
20
21. âIt ainât what you donât know that gets you into
trouble, itâs what you know for sure that just ainât
so.â â Mark Twain
21
22. âIt ainât what you donât know that gets you into
trouble, itâs what you know for sure that just ainât
so.â â Mark Twain (although thereâs no evidence that
Twain actually said this)
22