This talk takes as inspiration Prof. Carole Goble's notion that research communication should be more like software development practices. It looks at some of the state of the art and how it fits into to that framework. It argues that we are moving towards that vision and discusses some of the norms that need to be accepted in this new world. Presented at http://www.dagstuhl.de/15302

1. Paul Groth
Elsevier Labs
@pgroth | pgroth.com
“Don’t Publish, Release”
Revisited
July 19 – 24 , 2015, Dagstuhl Perspectives Workshop 15302
Digital Scholarship and Open Science in Psychology and the Behavioral Sciences
http://www.dagstuhl.de/15302

2. “All studies would be adequately powered; exploratory and
confirmatory research would be clearly distinguished, and the
latter would be subject to pre-registration; study materials
and raw data would be archived for examination and use by
others; and replications would be highly valued and common-
place. In reality, time and money are limited.”
“Unfortunately, there has been a recent “crisis in
confidence” among psychologists about the
quality of psychological research (Pashler &
Wagenmakers, 2012).”

3. • All empirical papers must archive their data upon acceptance in order to be published unless the authors provide a
compelling reason why they cannot (e.g., expense, confidentiality). The action editor will be the final arbiter of whether the reason is
sufficiently compelling.
• “Data” refers to an electronic file containing nonidentified responses that are potentially already coded. Normally, the data would represent
an early stage of electronic processing, before individual responses have been aggregated. The data must be in a
form that allows all reported statistical analyses to be reproduced
while retaining the confidentiality of individual participants. This entails that the data are formatted and documented in a way that makes
the structure of the data set readily apparent.
• Archiving consists either of submitting the data to the journal (to be displayed as supplementary material at the end of the article), sending
it to some other archive that is accessible to established researchers and maintained by a substantial established institution, or authors
making the data available on their own website, assuming that they can assure us the site will be maintained by a recognized institution for
a reasonable period of time. Again, action editors will be the final arbiters of the appropriateness of an archive.
• Any publication that reports analyses of or refers to archived data will be expected to cite the original
publication in which the data were reported.
• This policy is new and therefore open to modification. Our aim is to implement a policy that maximizes transparency while minimizing the
burden on authors.

4. http://saravanerp.com

5. https://www.force11.org/meetings/beyond-pdf-2

6. Don’t Publish. Release!
Professor Carole Goble FREng FBCS
University of Manchester, UK
carole.goble@manchester.ac.uk

7. Research
Components:
data, codes, interpretation
Forks
Changes
Versions
Builds on
prior work

8. Don't publish. Release!
Treat ALL Components and ALL Research Like Software

9. So where are we at?

11. Registered Reports
• “conducting the peer review prior to data
collection and analysis”

15. http://ivory.idyll.org/blog/replication-i.html

17. PDF DOWNLOADS ONLY?

19. Is anything still hard?

20. 20Yolanda GilUSC Information Sciences Institute gil@isi.edu
Measuring Time Savings with
“Reproducibility Maps” [Garijo et al PLOS CB12]
2 months of effort in reproducing published method (in PLoS’10)
Authors expertise was required
Comparison of
ligand binding
sites
Comparison of dissimilar
protein structures
Graph network
generation
Molecular Docking
Work with D. Garijo of UPM and P. Bourne of UCSD

21. 21Yolanda GilUSC Information Sciences Institute gil@isi.edu
“The Reproducibility Project: Cancer Biology has been funded through a
$1.3mm grant from the Laura and John Arnold Foundation,”
http://validation.scienceexchange.com/#/cancer-
biology

22. Cost of documentation
http://www.indoition.com/en/services/costs-
prices-software-documentation.htm

23. Two norms sum it up
• Be transparent*
• Embrace the iteration inherent in science
* If we encourage transparency, well then we should be constructive

24. Constructive criticism
“I like to think of the Braintrust as
Pixar’s version of peer review, a
forum that ensures we raise our
game – not by being perspective
but by offering candor and deep
analysis”
-- Ed Catmull p. 93