a panel talk for the 1st Workshop on E-science ReseaRch leading tO negative Results (ERROR), held in conjunction with the 11th eScience conference on 3 September 2015 in Munich, Germany

1. www.ci.anl.gov
www.ci.uchicago.edu
Scientific research:
What Anna Karenina teaches us
about useful negative results
Daniel S. Katz
Senior Fellow, Computation Institute (University
of Chicago & Argonne National Laboratory)
Affiliate Faculty, CCT (LSU)

2. www.ci.anl.gov
www.ci.uchicago.edu
2 Parallel and Distributed Application Paradigms
• "Happy families are all alike; every unhappy
family is unhappy in its own way.”
-- Leo Tolstoy. Anna Karenina

3. www.ci.anl.gov
www.ci.uchicago.edu
3 Parallel and Distributed Application Paradigms
Anna Karenina principle
• Describes an endeavor in which a deficiency in any one of a
number of factors dooms it to failure
• Consequently, a successful endeavor (subject to this principle)
is one where every possible deficiency has been avoided
• In statistics, the term is used to describe significance tests:
there are any number of ways in which a dataset may violate
the null hypothesis* and only one in which all the
assumptions are satisfied
-- https://en.wikipedia.org/wiki/Anna_Karenina_principle
*The null hypothesis usually refers to a general statement or
default position that there is no relationship between two
measured phenomena, or no difference among groups
-- Everitt, Brian (1998). The Cambridge Dictionary of Statistics

4. www.ci.anl.gov
www.ci.uchicago.edu
4 Parallel and Distributed Application Paradigms
Scientific research process
• The goal of research is provable (reproducible)
knowledge
– Expressed descriptively
o Something an observer can read, i.e., in a paper, book,
algorithm, etc.
– Expressed physically
o Something an observer can touch, measure, etc., i.e., an
experiment
– Expressed digitally
o Something an observer can run on a computer, i.e., simulation
or analysis software and required data
• Both physical and digital results usually also require
descriptive analysis

5. www.ci.anl.gov
www.ci.uchicago.edu
5 Parallel and Distributed Application Paradigms
The value of research results
• How do we decide if there is value in a result?
– Results can be positive or negative
– Must be new (and arguably provable)
– Positive results that show something novel
– Positive results that show something already in a
novel way

6. www.ci.anl.gov
www.ci.uchicago.edu
6 Parallel and Distributed Application Paradigms
The value of negative research results
• Many results can show that something doesn't
work
• First demonstration that something doesn’t
work is novel
– May be sufficient for publication by itself
– May require understanding of why
• Otherwise, want to provide new understanding
of why it doesn't work
– Potentially leading to possible paths to make it work

7. www.ci.anl.gov
www.ci.uchicago.edu
7 Parallel and Distributed Application Paradigms
Back to Anna Karenina
• "Happy families are all alike; every unhappy
family is unhappy in its own way.”
-- Leo Tolstoy. Anna Karenina
• Lesson
– To sell a story, it needs to be novel, perhaps by
explaining why the family is unhappy