20121214 ICERM reproducibility conference

1. tools for communicating in the
computational sciences
Brian M. Bot | Senior Scientist | Sage Bionetworks
clearScience
14 December 2012

2. Sage Bionetworks

3. Sage Bionetworks
I love Sage notebooks!

4. a non-profit organization which pilots a
variety of components that are necessary
to build a scientific research “commons”
why?
Sage Bionetworks

5. “We Must Guard Against
the acquisition of unwarranted influence,
whether sought or unsought, by the
Military Industrial Complex”
- Dwight D. Eisenhower 1961Medical

6. not conducive for a ‘commons’

7. institutional incrementalism
individual tenure
proprietary short term solutions
not conducive for a ‘commons’

8. commonsenabling a
open data
accessible platform
clear communication

9. “The problem is that right now, it’s not easy to donate your
data to health research.”
“The goal of Consent to
Research is to play a part in the
transformation of health from
something we experience
passively to something we
experience actively.”
http://weconsent.usJohn Wilbanks, Chief Commons Officer
open data

10. open data
accessible platform
clear communication
commonsenabling a

11. ‣ compute
‣ hardware
‣ software
‣ data
‣ code
analysis environment
RESTful APIs
accessible platform

12. open data
accessible platform
clear communication
commonsenabling a

13. clear communication

14. Deception at Duke

15. research scandals represent
merely the extreme of a
continuum in the culture of
academic research

16. the status quo tolerates
poor communication
of findings
6%
21%
8%
11%
54%
cannot
reproduce
can
reproduce in
principle
can reproduce
w/discrepancies
can reproduce from
processed data w/
discrepancies
can
reproduce
partially
Ioannidis A. et al. Repeatability of published microarray gene expression analyses. Nature Genetics 41, 149-155 (2009) | doi:10.1038/ng.295

17. 208,294,724
datapoints
124 pages
supplemental material
?? lines
unobtainable source code
?? version or architecture of
statistical analysis program (R)
enumerable R packages
and package dependencies
key R package “ClaNC”
no longer available
442 citations
often what is in principle
reproducible, is not
practically reproducible
unidentified publication
‣ from journal with 5 year impact factor of 28
‣ article freely available for download
‣ data freely available for download

18. how are we to move science forward
if we cannot understand what was done previously?

19. let’s go back to basics

20. 4. test hypothesis experimentally
5. analyze experimental data
7. publish results
6. draw conclusions based on data
scientific method
1. define a question
2. gather information and resources (background research)
3. form a hypothesis
8. retest (frequently done by other scientists)
4. test hypothesis experimentally
5. analyze experimental data
7. publish results
6. draw conclusions based on data

21. 7. publish results

22. finitein
∞
...

23. submit to journal
analyze on local
machine
write a document
sent to
reviewers
as pdf
printed
on paper
static html
representation
experimentally
generate data
accepted &
digitally
typeset
static pdf
representation
store on local
server

24. are being artificially uncoupled from
scientific claims
science itself

25. is hardscience

26. is hardcommunication
(especially for scientists)

27. clearScience
re-imagining scientific communication
allow consumption of content at a
variety of levels of complexity
and abstraction
leverage Synapse RESTful APIs

28. clearScience
allow consumption of content at a
variety of levels of complexity
and abstraction
“hand the keys over” to the reviewers

30. scientific communication
needs to evolve

31. along with science
needs to evolve

32. make it easy to do
good science

33. clearScience
make it easy to do

34. Acknowledgements
Sage Bionetworks
David Burdick - Rockstar Engineer
Stephen Friend - President and CEO
Erich S. Huang - Director of Cancer Research
Mike Kellen - Director of Technology
External Partners
Myles Axton - Nature Genetics
Phil Bourne - PLoS Computational Biology
Josh Greenberg - Alfred P. Sloan Foundation
Kelly LaMarco - Science Translational Medicine
Eric Schadt - Mount Sinai School of Medicine