Talk prepared for motivating the Session proposed by AMIGA team to SKA Office and organized by William Garnier (SKAO) for ESOF (European Science Forum) held in Toulouse in July 2018

1. Is the current measure of
excellence perverting Science?
A Data deluge is coming, it is time to act
Lourdes Verdes-Montenegro
Instituto de Astrofísica de Andalucía (IAA-CSIC)
Session Organiser: William Garnier (SKAO)
(Submitter and Manager)
Theme #3 Science policy and transformation of research practice

2. ANY PROBLEM WITH THE SCIENTIFIC METHOD??
Scientific Reproducibility is a fundamental principle of the Scientific
Method, a process established in the 17th century that marked the
beginning of modern science and laid the foundations for the Philosophy
of Science.
We all agree, “reproducibility is great!”, right?
Then… What is the problem?

3. ANY PROBLEM WITH THE SCIENTIFIC METHOD??
Scientific Reproducibility is a fundamental principle of the Scientific
Method, a process established in the 17th century that marked the
beginning of modern science and laid the foundations for the Philosophy
of Science.
We all agree, “reproducibility is great!”, right?
Then… What is the problem?
Paradoxically:
• part of the scientific community claims that reproducibility is already
achieved (because they have a section describing their methods in
their papers, or because they share their data)
• the remainder mostly consider it a utopy

4. Questionnaire on reproducibility (1500 scientists)
• 70% of researchers have tried and failed to
reproduce another scientist's experiments
• > 50% have failed to reproduce their own ones!
• Chemistry: 90% (60%)
• Biology: 80% (60%)
• Physics and engineering: 70% (50%)
• Medicine: 70% (60%)
• Earth and environmental science: 60% (40%)
ANY PROBLEM WITH THE SCIENTIFIC METHOD??
ACTUALLY, YES!
2016

5. Questionnaire on reproducibility (1500 scientists)
• 70% of researchers have tried and failed to
reproduce another scientist's experiments
• > 50% have failed to reproduce their own ones!
• Chemistry: 90% (60%)
• Biology: 80% (60%)
• Physics and engineering: 70% (50%)
• Medicine: 70% (60%)
• Earth and environmental science: 60% (40%)
ANY PROBLEM WITH THE SCIENTIFIC METHOD??
ACTUALLY, YES!
2016
Aha! So you don’t empathise?

6. Questionnaire on reproducibility (1500 scientists)
• 70% of researchers have tried and failed to
reproduce another scientist's experiments
• > 50% have failed to reproduce their own ones!
• Chemistry: 90% (60%)
• Biology: 80% (60%)
• Physics and engineering: 70% (50%)
• Medicine: 70% (60%)
• Earth and environmental science: 60% (40%)
ANY PROBLEM WITH THE SCIENTIFIC METHOD??
ACTUALLY, YES!
Maybe with this?
2016

7. • Reproducibility (theoretical definition): An experiment/study is reproducible if an
external researcher could repeat the same procedures and confirm the results using the
same set up, input data and methods
• Reproducibility (in practice): input data, methods, set up parameters, output data and
results, and the computational environment, together with details on the context and links
between the pieces of the experiment.
UHMMM,WHAT DOYOU MEAN BY REPRODUCIBILITY?
Moving from narratives
(last 300 yrs) to the
actual output of research

8. • Reproducibility (theoretical definition): An experiment/study is reproducible if an
external researcher could repeat the same procedures and confirm the results using the
same set up, input data and methods
• Reproducibility (in practice): input data, methods, set up parameters, output data and
results, and the computational environment, together with details on the context and links
between the pieces of the experiment.
UHMMM,WHAT DOYOU MEAN BY REPRODUCIBILITY?
BTW: Reproducibility is
not the aim, is the mean

9. PERSPECTIVES

10. PERSPECTIVES
Data to the desktop: “individual scientist”
• I have the best code, which I know how to use and can do special things
• I do not trust any “pipeline” that you made
• partly because I know better how to do it
• partly because I read the news and there is a reproducibility crisis
• well, and I can hardly reproduce the results of my own papers
some years later...
• In general I want full control of the software and of the computational
environment

11. PERSPECTIVES
Computation to data, providers perspective: Data Centres
Mandatory in the Era of
Megascience infrastructures

12. PERSPECTIVES
Computation to data, providers perspective: Data Centres
• We need to install your software in our platform. Can we trust it? Can we run it?
Environment, dependencies, etc
• Hey, we are offering services to the community, computation + tools. We would
be grateful if you allow us to share it with other users (with proper credit)
• Mmmm, sharing is great, but, putting the software in the platform is not
enough: you need to provide the context for people to be able to rerun the
software on the same or other data
Mandatory in the Era of
Megascience infrastructures

13. PERSPECTIVES
Large alliances of scientists to develop Key Science Projects
Mandatory (as well) to analyse the data
deluge from this next generation of facilities

14. PERSPECTIVES
Large alliances of scientists to develop Key Science Projects
• We have tools to generate Advanced Data Products, and we will put them there
where the storage and computation is (Data Centres)
• But... we put effort on it, what would we gain if we make the *additional effort*
to make it reusable? If we make it, then we will pave the way to competitors
• Well, maybe we will share in 4 yrs time (PhD typical time)
Mandatory (as well) to analyse the data
deluge from this next generation of facilities

15. PERSPECTIVES
Publishers
• Will we need different profiles of referees to evaluate the scientific discussion
together with the data quality and the methods (aka. Reproducibility)?
• If the data and the methods (tools) will be in Data Centres, will our referees need
to become a “user” of the Data Centres to be able to validate a paper?
• Will we be able to engage so many referees as may be needed?
• Will we need to validate the data, the tools, and the scientific analysis separetely?
The challenge of going “beyond the PDF”

16. PERSPECTIVES
Policy makers / funding agencies
• How to measure reproducibility?
• How to weight it and/or aggregate with other indicators?
• Is it affordable / sustainable?
Reproducibility as a key element of Open Science

17. METRICS
... “Science is being
killed by numerical
ranking,”[...] Ranking
systems lures scientists
into pursuing high
rankings first and good
science second
Productivity seems to prevail
over Discovery

18. METRICS
... “Science is being
killed by numerical
ranking,”[...] Ranking
systems lures scientists
into pursuing high
rankings first and good
science second
Reproducibility
crisis

19. SKA1: 197 dishes + 125.000 dipoles
SKA: 2500 dishes + 500.000 dipoles
THE SQUARE KILOMETRE ARRAY
Ø 1000 scientists &
engineers from > 270
institutions, > 20 countries.
Ø 11 Member countries

20. SKA1: 197 dishes + 125.000 dipoles
SKA: 2500 dishes + 500.000 dipoles
THE SQUARE KILOMETRE ARRAY
Ø 1000 scientists &
engineers from > 270
institutions, > 20 countries.
Ø 11 Member countries

21. THE SQUARE KILOMETRE ARRAY
The Challenge: Extraction of scientific knowledge
• Direct delivery to end users is unfeasible
• International distributed scientific teams
SKA Regional Centres will provide access to SKA data products, tools and
processing power to generate and analyse Advanced Data Products

22. AREWE READY?
• We are in a race to exploit ever larger datasets:
in our quest for “efficiency” we risk forgetting about reproducibility
• Unless we are ready to change the way in which we, the scientists, work, there
is no guarantee that the quality of Science will improve.

23. AREWE READY?
The era of Big Data is beginning across sciences
Today is the time to ask what kind of Research mega-science
infrastructures want to do in tomorrow’s future
Are we ready to take up that challenge?
• We are in a race to exploit ever larger datasets:
in our quest for “efficiency” we risk forgetting about reproducibility
• Unless we are ready to change the way in which we, the scientists, work, there
is no guarantee that the quality of Science will improve.