This presentation is about a lecture I gave within the "Green Lab" course of the Computer Science master, Software Engineering and Green IT track of the Vrije Universiteit Amsterdam: http://masters.vu.nl/en/programmes/computer-science-software-engineering-green-it/index.aspx http://www.procaccianti.me

2. Vrije Universiteit Amsterdam
Hypothesis: a formal statement about a phenomenon
● Null hypothesis H0
: no real trends or patterns in the
experiment setting (aka ~Q)
● Alternative hypothesis Ha
: there are real trends or
patterns in the experiment setting (aka Q)

3. Vrije Universiteit Amsterdam
● We aim at verifying the null hypothesis (~Q)
○ we test the null hypothesis H0
We can reject the null hypothesis → we can draw conclusions
This comes from Popper (1959): any statement in a scientific
field is true until anybody can contradict it
● Aiming at verifying Q is WRONG
○ Provides no insight on the conjecture

4. Vrije Universiteit Amsterdam
●
●
●

5. Vrije Universiteit Amsterdam

6. Vrije Universiteit Amsterdam
●
○
"P-value in statistical significance testing" by User:Repapetilto @ Wikipedia & User:Chen-Pan Liao @ Wikipedia -
https://en.wikipedia.org/wiki/File:P_value.png. Licensed under CC BY-SA 3.0 via Wikimedia Commons

7. Vrije Universiteit Amsterdam
●
●

8. Vrije Universiteit Amsterdam
●
○
○ α
●
○
○ β

9. Vrije Universiteit Amsterdam

10. Vrije Universiteit Amsterdam
●
○ β
●

11. Vrije Universiteit Amsterdam

12. Vrije Universiteit Amsterdam
● α
○ α
○
● β
○
○
●

13. Vrije Universiteit Amsterdam
●
●
●
○
○
■ α

14. Vrije Universiteit Amsterdam
●
●
○
●
○
○

15. Vrije Universiteit Amsterdam
α
0.38

16. Vrije Universiteit Amsterdam
α
● α
○
●
0.38
α

17. Vrije Universiteit Amsterdam
●
○
●
○
○
●
○

18. Vrije Universiteit Amsterdam
●
●

19. Vrije Universiteit Amsterdam
●
○
●

20. Vrije Universiteit Amsterdam
● Analysis and Interpretation: Chapter 10
Experimentation in SE, Wohlin et al. (Blackboard)
● Hypothesis Testing: Chapter 10, Section 3
Experimentation in SE, Wohlin et al. (Blackboard)
● "It's the effect size, stupid" (Coe, R.)
● Plenty of online resourcestutorials

21. Vrije Universiteit Amsterdam
● Coin toss example and other content from Empirical
Methods in Software Engineering, Marco Torchiano,
Politecnico di Torino - http://softeng.polito.it/EMSE/
● Effect Size from Basics of Empirical Software Engineering,
Massimiliano Di Penta, University of Sannio, Italy

22. Vrije Universiteit Amsterdam
g.procaccianti@vu.nl
i.malavolta@vu.nl