What use are experiments for industry? 1. Experiments are a powerful tool for evaluating technologies. Experiments provide an objective comparison rather than subjective opinions 2. The results of experiments should allow industry to: Base decisions on objective grounds and increase control on software development

1. Natalia Juristo
University of Oulu
&
Universidad Politécnica de Madrid

2. FiDiProFinland Distinguished Professor Program
Project start, January 2013
Experimental Software Engineering
Industrial Laboratory (ESEIL)

4. MISSION
!"#$%#$&#'()%*$%)(+%!"#$$%&'%(%)*%,*$-.,)%
+/0+1#2+$)&%#$%)(+%&*34"1+%#$-.&)15%
%
6$,1+"&+%&*34"1+%#$-.&)15%"4"1+$+&&%
"7*.)%)(+%)(%*)$&%((%*8%+/0+1#2+$)&%%
9.&(%8*14"1-%)(+%%+,%-./%&0#$1,#-#2.'/1
"00:#+-%)*%3&45$%2'%1'%&%-#64&1"7*.)%
&*34"1+%-+;+:*02+$)%

6. !"#$%#&'$()*$'+$,-.$*+/0&#12$
3&$'+$1+&#$*+/0&#12$3-$'"#$
&#0435#$+2$+'"#0&
6)117)08$+2$'"#$&53#-9,5$7#'"+.$'+$
:#-#0)'#$"3:"$;/)13'*$$
8-+(1#.:#$
<+-./59-:$#=>#037#-'&$%03-:&$'"#$
&53#-9,5$7#'"+.$'+$?+@()0#$
A-:3-##03-:$

7. !"#$%&'$()*+&*+),$+-(./+01/++
)-+%$2$1.+314*$5$6$3)+%$.17-(*,&#*+
8(9$%*)1(9+),$+,&99$(+'$3,1(&*'*+137(:+
;$,&(9+),$+;$,12&-%+0$+-;*$%2$+
<9$(7=/+!"#$"%&'(+#.1/&(:+1+%-.$++
&(+*->01%$+9$2$.-#'$()+1(9+),$++
)*++'),*+(+;$)0$$(+),$'+

8. !"#$"%%&$"#'($)*$+,$"%)'
!"#$"%&'($)(*+(),-.%/$0&"s and market-speak to
1%($2"&*$".'$&3)"*,"4$"."4#%&%$".'$&.2-*3",-.$
Fact-based management
-*$%"./*0,,1'20,$('/"($"#)'
)345,('3%,+'64'$(%".71'
5))4&+"0&4&6#$+-.'&,-.%/$7("8.(%%(%$".'$%6*(.96:%$
-2$%-;7"*($'(<(4-)3(.6$6(+:.-4-9&(%$
$
=10>(+,<($-)&.&-.%$+".$4("'$6-$3&%6"8(.$
)(*+(),-.%$-2$*("4&6#$

9. What Use Are Experiments for Industry?
!"#$%&'$()*+,%$+,+#-.$%/01+)--1+/-%+$2,10,3(4+
)$56(-1-4&$*+
7 !"#$%&'$()*+#%-2&8$+,(+-9:$532$+5-'#,%&*-(+%,)6$%+)6,(+*09:$532$+
-#&(&-(*+
;6$+%$*01)*+-/+$"#$%&'$()*+*6-018+,11-.+&(80*)%<+)-+
7 =,*$+8$5&*&-(*+-(+-9:$532$+4%-0(8*+
> ?$*01)*+/%-'+,(+$"#$%&'$()+,%$+2,1&8+,)+5-'#,(<+1$2$1+
7 @(5%$,*$+5-()%-1+-(+*-A.,%$+8$2$1-#'$()+
> @8$(3/<+%$1$2,()+2,%&,91$*+-/+*-A.,%$+)$56(-1-4&$*+
1.
2.

11. What an Experiment Is Not
!"#$%#&'$#()#&*'#+%$,"-./0$!"#$%&$'"!()*&+$$
1*%"$-%"#&$,*'*/2&$%#&',$
$
3+#4,"-%$#()#&*#+5#$
6 78$-+5#$1#+%$92//--+*+:;$
6 <2++-%$9#$:#+#&2/*=#0$%-$-%"#&$02>,?$-%"#&$)#-)/#$
@.9A#5BC#$&#)#2%#0$#()#&*#+5#$
6 7D2./$"2,$2$/-%$-E$#()#&*#+5#$'2+2:*+:$)&-A#5%,;$
6 <2+$:#+#&2%#$&./#,4-E4%".'9,?$*,$5-+F+#0$%-$D2./?$*,$,.9A#5BC#$
<-//#5%#0$02%2$E&-'$)&-A#5%,$
6 73.&$0#)2&%'#+%$,%-&#,$02%2$E&-'$2//$)&-A#5%,;$
6 8+$&#2/$)&-A#5%,?$C2&*29/#,$52++-%$9#$5-+%&-//#0$2,$+#5#,,2&>$%-$.+0#&,%2+0$1"2%$%"#$
52.,#,$2+0$#G#5%,$2&#$$

12. A SE Experiment
Task i
A
B
M1
M1
Task i
M2
M2
Software process
Software process
Task i
Task i
Task i
Task i

14. What is Needed to
Conduct an Experiment?
! "#$%&%'()%*+#
, -.%#/(*%#$%&%'()%*+0#1.%#/(*%#*%'234'%#*%+5'1+#
! 6&%*#780#29#)(++24'%#
! "#$%&%'()%*+#5:$%*#1.%#+3/%#;(:$2<(:+#
, =(:>:%$#2:#3#*((/0#5+2:?#+3/%#2:9*3+1*5;15*%0#@#
! "#$%&%'()%*+#$(2:?#1.%#+3/%#13+A#5:$%*#+15$B#
, C#13+A#1B)2;3''B#13A%+#31#'%3+1#1D(#.(5*+#1(#;(/)'%1%#
! E%)%:$2:?#(:#1.%#%F)%*2/%:13'#$%+2?:#D%#;3:#+)'21#)3*<;2)3:1+#(*#%&%*B4($B#
$(2:?#21#1D2;%#

16. Train people in
exchange
Researchers provide free training
Companies provide participants for the experiment

17. Our TDD Experiment
TRAINING EXERCISES EXPERIMENTAL TASKS
DAY 1 Reminder of
Unit Testing
concepts
3 Exercises to
warm up
CONTROL TASK (Test-last)
DAY 2 Test-driven
Development
3 TDD Exercises
DAY 3 2 TDD Exercises TREATMENT TASK (TDD)

19. Quality
@ F-Secure TDD Experiments

20. • The mean for ITL is 54% of
correctly implemented user
stories
• The mean for TDD is 76%
• Subjects applying
TDD get a 17%
improvement in
quality compared
to subjects
applying ITL
Quality
@ F-Secure TDD Experiments
• The observed
difference
between ITL
and TDD is not
significant
EFFECT SIZE
HYPOTHESIS
TESTING

21. Productivity
@ F-Secure TDD Experiments

22. • The mean for ITL is 23%
• The mean for TDD is
greater at 48%
• Subjects applying TDD
show a 24%
improvement in
productivity compared
to subjects applying
ITL
• The observed
differences between
ITL and TDD is
significant
Productivity
@ F-Secure TDD Experiments

23. Other Data Analysis

24. Dissecting TDD: Post-hoc analysis
! "#$%&'()#$*#+,-.*/.,011,%2%3%#(,3*+.(,4%,(.%,)/56%,7&*#/*72%,
/)8'*#+,*37&96%$,:8)2*(;,
< =)#9,/;/2%',
< 0%'(,>&'(?,(.%,/9$%,
< @%A)/(9&*#+,)B%&,&)(.%&,(.)#,$%'*+#,>&'(,
! =)#9C$%6%2973%#(,.)',39'(,4%)&*#+,9#,*37&96%$,:8)2*(;,,,
< 0*#;,DC3*#8(%,$%6%2973%#(,/;/2%',
< E,8#*(,*#/&%)'%,*#,:8)2*(;,A9&,%6%&;,FGDH,&%$8/59#,*#,/;/2%,'*I%,
! =%%$,(9,4%,/9#>&3%$,-*(.,)#,%J7%&*3%#(K,

25. Schedule: Our TDD Experiment
TRAINING EXERCISES EXPERIMENTAL TASKS
DAY 1 Testing tool
Concepts
2 Tool Exercises
1 Motivational
exercise (ITL)
BASELINE TASK (Your Way)
DAY 2 Slicing 2 Slicing Exercises CONTROL TASK (ITL)
DAY 3 Test-driven
Development
3 TDD Exercises TREATMENT TASK (TDD)

27. Replications of Our TDD Experiment
!"#$%&'$()*+,-$.&/(,01,#%$2#3.),,
4 5*6*,7,58(&),.$9(/,
: ;2<$=>%$,?@,.&)$.A,2B,CD,#*%E=&#*().,
: 8FG,?<#*&(A,2B,@0,#*%E=&#*().,
4 HII,7,JK$.),.$9(/,
: !+$L)%3M&),2B,N,#*%E=&#*().,
!"#$%&'$()*+,-$.&/(,C1,=%3..,7,.+&=&(/,)%*&(&(/,
4 5*6*,7,58(&),.$9(/,
: H3'#*(O,P,?!.)3(&*A,
: 8FQ,?<#*&(A,
: !<F!,?!=>*R3%A,
!"#$%&'$()*+,-$.&/(,@1,!-C,7,S3>%T*O,=3()%3+,)%$*)'$(),,
4 HII,7,U(O,.$9(/,
: !%&=..3(,
4 FO)V3(,7,WW,
: ;2<$=>%$,

28. Quality: TL vs TDD
F-Secure Elektrobit Univ. Politecnica Valencia
TL < TDD (42%)
Java
31 Students
No need more subjects
TL = TDD (32%)
C++
9 Professionals
Need more subjects
TL = TDD (17%)
Java
24 Professionals
No need more subjects

29. Productivity: ITL vs TDD
ITL < TDD (41%)
Java
31 Students
No need of more subjects
ITL < TDD (21%)
C++
9 Professionals
No need of more subjects
ITL < TDD (24%)
Java
24 Professionals
No need of more subjects
F-Secure Elektrobit Univ. Politecnica Valencia

30. !"#$$%$&'#$%()*+,+-'+.%
/)012%3+%(4+%2&%(+5+$&6+,.7%286+%&9%.&:;#,+<%
=>>%$+#(.%2&%)"6,&5+"+-2.%)-%?4#$)28%#-(%
6,&(4'@5)28%
A&-B.)0-)C'#-2%()*+,+-'+.%9&,%?4#$)28%
!)0-)C'#-2%()*+,+-'+.%9&,%6,&(4'@5)28%
%
!2#3$+%,+.4$2.%9&,%()*+,+-2%)-9,#.2,4'24,+.%
D#5#%E%FGG%
DH-)2%E%I2+.2%
%

31. We Are Now Recruiting !
! "#$%&#$'(()*+,$-(&$.(/0%+*#1$*+2#&#12#3$*+$&#.#*4*+,$(5&$.(5&1#$
(+$677$
8 "#$9*''$&5+$2:#$#;0#&*/#+2$%1$0%&2$(-$2:#$2&%*+*+,$
<&##$=$3%>1$2&%*+*+,$(+$677$
?(5&$'(.%'$#;0#&*/#+2$&#15'21$
@*3#$50$2($<%.2AB%1#3$C%+%,#/#+2$
natalia.juristo@oulu.fi

32. Natalia Juristo
University of Oulu
&
Universidad Politécnica de Madrid
Email: natalia@fi.upm.es