A talk about improving reproducibility, simplifying genomic machine learning, and using the resulting predictors to improve power in clinical trials.

1. fixing the leaks in the genomics

2. http://jhudatascience.org/

3. https://www.coursera.org/specialization/genomics/41

4. @simplystats
http://simplystatistics.org

5. @jtleek
http://www.jtleek.com

6. https://www.counsyl.com/

7. Their basic pitch was
“Genomics is a fraud”
“
”
http://www.technologyreview.com/news/535771/a-contrarian-in-biotech/

9. “The explosive growth of next-generation
sequencing data submitted into the SRA
exceeds the growth rate of storage
capacity
”
http://www.ncbi.nlm.nih.gov/pubmed/22009675

13. 3cost
analyst variation
motivation

14. 1cost

15. costs
money
interpretability

16. http://arxiv.org/pdf/math/0606441.pdf

17. http://www.ncbi.nlm.nih.gov/pubmed/19276151

19. @leekgroup

21. http://www.ncbi.nlm.nih.gov/pubmed/25788628

22. http://www.ncbi.nlm.nih.gov/pubmed/25788628

23. Agilent/Grade 1 Agilent/Grade 3 Illumina/Grade1 Illumina/Grade3
100%
75%
50%
25%
0%
Accuracy
Pam Scaled Pam Unscaled TSP
http://www.ncbi.nlm.nih.gov/pubmed/25788628

24. algorithm
1.select useful pairs
2.screen pairs for association
3.build a simple cart predictor

25. http://www.ncbi.nlm.nih.gov/pubmed/19276151

26. Patil et al. (in prep)

27. Patil et al. (in prep)

28. Patil et al. (in prep)

29. @leekgroup
Data:
xik
- value for feature i, sample k
yk
- group indicator for sample k
TSP is (i,j) pair that maximizes:
|Pr(xik
< xjk
| yk
=1) – Pr(xik
< xjk
| yk
=0)|⌃ ⌃
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1989150/

30. @leekgroup
zijk
=1(xik
< xjk
)
E[zijk
|yk
] = a0ij
+ a1ij
yk
→ max |a1jk
| = TSP
Patil et al. (in prep)

31. @leekgroup
• Not the same as TSP
• But |â/s.e.(â)| = |û/s.e.(û)|, algebraically
• “Variance regularized” TSP
• zijk
invariant to monotone transformations
• Fix parameters → find features
E[yk
|zijk
] = u0ij
+ u1ij
zijk
Patil et al. (in prep)

32. @leekgroup
1. Calculate t-statistic for all pairs
2. Choose top pair (or covariate)
3. Continue for a fixed number of pairs
E[yk
|zijk
] = u0ij
+ u1ij
zijk
Patil et al. (in prep)

33. @leekgroup
http://astor.som.jhmi.edu/~marchion//breastTSP.html

34. @leekgroup
USP7 < RP11-423C15.3
NM_018610 < MTCH1
RND1 < LGALS14
No
Recur
No
Recur
No
Recur
Recur
No Yes
No Yes
No Yes

35. @leekgroup

36. @leekgroup
Mammaprint
Patil et al. (in prep)

37. 2analyst variation

41. what went wrong?
2things

42. what went wrong? transparency
The data/code weren’t reproducible

43. what went wrong? transparency
There was a lack of cooperation

44. what went wrong? expertise
They used silly prediction rules
(Pr(FEC) = 5/8[Pr(F) + Pr(E) + Pr(C)] – ¼)

45. what went wrong? expertise
They had study design problems
(Batch effects)

46. what went wrong? expertise
Their predictions weren’t locked down
Today: Pr(FEC) = 0.8
Tomorrow: Pr(FEC) = 0.1

47. At the end of the day the Potti
analysis was fully reproducible
The problem is that the analysis
was wrong

48. @leekgroup
http://bit.ly/10vS1yt

49. @leekgroup
http://bit.ly/OgW3xv

50. @leekgroup
Drinkel et al. Oganometalics 2013

51. @leekgroup

52. @leekgroup

53. @leekgroup

54. @leekgroup

56. http://simplystatistics.tumblr.com/post/19646774024/laws-of-nature-and-the-law-of-patents-supreme-court

57. 3motivation

59. $(from reducing sample size)

60. basic idea
randomization isn’t perfect
“rebalance” with baseline covariates
improve estimator precision

63. Ack Math!!!!

64. Estimate probability of being in arm given baseline covariates

65. Calculate initial estimate for each person using each arm
model using propensity score weighted logistic regression

66. Define a covariate as the residual from fitting the arm-level
models minus the arm-level means and fit new propensity
models

67. Use these propensities to re-fit WLR from (2), then average
predictions to get covariate-adjusted treatment effect

68. @leekgroup
http://astor.som.jhmi.edu/~marchion//breastTSP.html

69. @leekgroup
Age, Tumor Size, Grade 5.1%
Age, Tumor Size, Grade,
ER Status
4.9%
Mammaprint Risk
Category (MRC)
5.4%
Age, Tumor Size, Grade,
ER Status, MRC
7.8%

70. @leekgroup
Age, Tumor Size, Grade 5.1%
Age, Tumor Size, Grade,
ER Status
4.9%
Mammaprint Risk
Category (MRC)
5.4%
Age, Tumor Size, Grade,
ER Status, MRC
7.8%
Age, Tumor Size, Grade,
ER Status, TSP
6.2%

71. 3cost
analyst variation
motivation

72. acknowledgements
Leek group
Prasad Patil
Leo Collado Torres
Abhi Nellore
Claire Ruberman
Jack Fu
Kai Kammers
Collaborators
Michael Rosenblum
Benjamin Haibe-Kains
P.O. Bachant-Winner
Roger Peng

73. Prasad Patil
http://www.biostat.jhsph.edu/~prpatil/

74. Links
https://github.com/leekgroup/sig2trial
http://jtleek.com/talks/