Some subjective recommendations of how to proceed with experiments

1. Steffen Staab
staab@uni-koblenz.de
1WeST
Web Science & Technologies
University of Koblenz ▪ Landau, Germany
Experiments
Don‘t loathe them, but love them!
Steffen Staab

2. Steffen Staab
staab@uni-koblenz.de
2WeST
...a month in the lab can save an hour in the library...
Unknown source

3. Steffen Staab
staab@uni-koblenz.de
3WeST
...avoid the experiment
and
you may avoid finishing your thesis...
Steffen 2014

4. Steffen Staab
staab@uni-koblenz.de
4WeST
What we love and what we loath

5. Steffen Staab
staab@uni-koblenz.de
5WeST
Why do we loathe experiments?
Experiments
 require effort
 require time
 require money
 may be complicated
 may involve human subjects (BAD!)
 may even involve real people, i.e. not students (WORSE!!!)
 have unpredictable outcomes (THE WORST!!!!!)
 May have to be redone (THE NIGHTMARE!!!!!)

6. Steffen Staab
staab@uni-koblenz.de
6WeST

7. Steffen Staab
staab@uni-koblenz.de
7WeST
1 What do you talk about?
Define your key terms clearly!
Don‘t redefine existing terms!
Don‘t use figurative speech.
Negative example: Beer is a kind
of bread made from barley.
Define them independently of
how you compute them:
Negative example: We are doing this experiment to show that
Lakhsa is enjoyed by more people than Surströmming
Use the terms exactly as you defined them:
Negative example: The strömming must be enjoyed cold.

8. Steffen Staab
staab@uni-koblenz.de
8WeST
2 Formulate a hypothesis
What is your experiment about?
Only possible to define after
key concepts are clear!
As precise as possible:
...enjoy more...?
vs
...would do immediately again, if given the choice....
Briefly say what you abstract from and why.
A clear hypothesis saves time!

9. Steffen Staab
staab@uni-koblenz.de
9WeST
3 Fail-safe hypothesis
Hypothesis that is always interesting,
no matter whether the experiment succeeds or fails
 comparing different methods is often interesting
 Be impartial / not biased in favor of your method
AND
write in this way
 not: „we want to show that our method is better“
BUT
„we want to find out whether fermentation makes the fish
more enjoyable for Swedish and non-swedish“
Tackling unpredictability!

10. Steffen Staab
staab@uni-koblenz.de
10WeST
4 Write before performing experiment
Write
 The experiment
 The description of the outcome
 Receive input/criticism on this writing
BEFORE
 you perform the experiment
Avoid having to redo experiment!

11. Steffen Staab
staab@uni-koblenz.de
11WeST
5 Reverse experiment planning
Lead question: What can make your experiment go wrong?
 May not be understood / handled by test subjects
 Do assume that someone can misunderstand something
 Keep your original data, video recordings if possible
 ...
Real people are somewhat
unpredictable!

12. Steffen Staab
staab@uni-koblenz.de
12WeST
6 More is better...
 Not just one data set
 Do not vary arbitrarily, but vary
on what may help to explore
your hypotheses
 Not just one baseline
 Especially not one that is very weak,
only because it is available
• Tackling the „hard baseline“ is not impossible
• Tackling the „hard baseline“ right away may be easier
than doing 5 rounds until the paper is accepted
More....is less time!

13. Steffen Staab
staab@uni-koblenz.de
13WeST
7 When more is worse
Grounding in theory
Counterexample:
„....we investigated all the 100680 parameter configurations....“
Experiments produce outliers!
 Statistical effects
 Normalization exists, but is very strict
Save effort!

14. Steffen Staab
staab@uni-koblenz.de
14WeST
8 When reporting: Focus the reader‘s eye
Issue: Experiments may be complex
 Hard to do
 Hard to understand for the reader
Measures:
 Do not overwhelm the reader
 Focus the reader‘s eyes
 Do not just show graphs/charts/etc.
 Each graph/chart should give an answer to a question
 Reader should know why he sees the chart
Manage complexity!

15. Steffen Staab
staab@uni-koblenz.de
15WeST
9 Sound experiment structure
Evaluation procedure must be independent from the method
you used-7should also be applicable for a large set of
competitors
1. Set up; i.e. data, conditions
2. Evaluation measure, baselines/competitors
3. Results
4. Discussion
Keep this order and do not discuss results before you have
described setup (maybe briefly in intro/abstract)

16. Steffen Staab
staab@uni-koblenz.de
16WeST
10 Be honest and ethical
 If there are issues, report about them
 Publish the methods
 Publish the software
 Publish the data

17. Steffen Staab
staab@uni-koblenz.de
17WeST
Disclaimer
 I have never eaten Surströmming
 I might have eaten Lakhsa in a Tibetan restaurant, but
don‘t remember it