1. Mutation Testing Hernán Wilkinson UBA - 10Pines hernan.wilkinson@gmail.com Nicolás Chillo UBA nchillo@gmail.com Gabriel Brunstein UBA gaboto@gmail.com

2. WhatisMutationTesting? Techniquetoverifythequality of thetests

3. WhatisMutationTesting? Verify Quality of… Verify Quality of… Source Code Tests Mutation Testing

4. How does it work?1st Step: Create the Mutant Mutation Process The Source Code The “Mutant” The Mutation “Operator”

5. Examples DebitCard>>= anotherDebitCard ^(type = anotherDebitCard type) and: [ number = anotherDebitCard number ] Operator: Change #and: by #or: CreditCard>>= anotherDebitCard ^(type = anotherDebitCard type) or: [ number = anotherDebitCard number ]

6. Examples Purchase>>netPaid ^self totalPaid – self totalRefunded Change #- with #+ Purchase>>netPaid ^self totalPaid + self totalRefunded

7. Why?How does it help?

8. How does it work?2nd Step: Try to Kill the Mutant A Killer tries to kill the Mutant! The “Mutant” All tests run  The Mutant Survives!!! The Test Suite A test fails or errors  The Mutant Dies

9. Meaning… The Mutant Survives  The case generated by the mutant is not tested The Mutant Dies  The case generated by the mutant is tested

10. Example: Themutantsurvives DebitCard>>= anotherDebitCard ^(type = anotherDebitCard type) and: [ number = anotherDebitCard number ] Operator: Change #and: by #or: DebitCard>>= anotherDebitCard ^(type = anotherDebitCard type) or: [ number = anotherDebitCard number ] DebitCardTest>>testDebitCardWithSameNumberShouldBeEqual self assert: (DebitCardvisaNumbered: 123) = (DebitCardvisaNumbered: 123).

11. Example: Themutantdies DebitCard>>= anotherDebitCard ^(type = anotherDebitCard type) and: [ number = anotherDebitCard number ] Operator: Change #and: by #or: DebitCard>>= anotherDebitCard ^(type = anotherDebitCard type) or: [ number = anotherDebitCard number ] DebitCardTest>>testDebitCardWithSameNumberShouldBeEqual self assert: (DebitCardvisaNumbered: 123) = (DebitCardvisaNumbered: 123). DebitCardTest >>testDebitCardWithDifferentNumberShouldBeDifferent self deny: (DebitCardvisaNumbered: 123) = (DebitCardvisaNumbered: 789).

12. Example: Themutantsurvives Purchase>>netPaid ^self totalPaid – self totalRefunded Change #- with #+ Purchase>>netPaid ^self totalPaid + self totalRefunded Purchase>>testNetPaid | purchase | purchase := Purchase for: 20 * euros. self assert: purchase netPaid = (purchase totalPaid– purchase totalRefunded)

13. Example: Themutantdies Purchase>>netPaid ^self totalPaid – self totalRefunded Change #- with #+ Purchase>>netPaid ^self totalPaid + self totalRefunded Purchase>>testNetPaidWithOutRefunds Renamed! | purchase | purchase := Purchase for: 20 * euros. self assert: purchase netPaid = (purchase totalPaid– purchase totalRefunded) Purchase>>testNetPaidWithRefunds | purchase | purchase := Purchase for: 20 * euros. purchase addRefundFor: 10 * euros. self assert: purchase netPaid = (purchase totalPaid– purchase totalRefunded)

15. Runthe test suite relatedtothechangedcode

16. Ifa test errorsorfailsKillsthemutant

17. Ifalltestsrun TheMutantsurvives

18. SurvivingMutants show nottestedcasesThe Important Thing!

19. MuTalk MutationTestingToolfor Smalltalk (Pharo and Squeak)

20. Demo

22. For each method m

23. For each operator o

24. Changes mAST using o

25. Compiles mutated code

26. Changes method dictionary

28. Remove #not

29. Replace #and: with #eqv:

30. Replace #and: with #nand:

31. Replace #and: with #or:

32. Replace #and: with #secondArgResult:

33. Replace #and: with false

34. Replace #or: First Condition with false

35. Replace #or: Second Condition with false

36. Replace #or: with #and:

38. Replace #'<=' with #<

39. Replace #'<=' with #=

40. Replace #'<=' with #>

41. Replace #'>=' with #=

42. Replace #'>=' with #>

43. Replace #'~=' with #=

44. Replace #< with #>

45. Replace #= with #'~='

46. Replace #> with #<

47. Replace #max: with #min:

49. Remove at:ifAbsent:

50. Replace #reject: with #select:

51. Replace #select: with #reject:

52. Replace Reject block with [:each | false]

53. Replace Reject block with [:each | true]

54. Replace Select block with [:each | false]

55. Replace Select block with [:each | true]

56. Replace detect: block with [:each | false] when #detect:ifNone:

57. Replace detect: block with [:each | true] when #detect:ifNone:

58. Replace do block with [:each |]

59. Replace ifNone: block with [] when #detect:ifNone:

60. Replace inject:aValueinto:aBlock with aValue

62. Replace #* with #/

63. Replace #+ with #-

64. Replace #- with #+

66. Remove Exception Handler Operator

67. Replace #ifFalse: receiver with false

68. Replace #ifFalse: receiver with true

69. Replace #ifFalse: with #ifTrue:

70. Replace #ifFalse:IfTrue: receiver with false

71. Replace #ifFalse:IfTrue: receiver with true

72. Replace #ifTrue: receiver with false

73. Replace #ifTrue: receiver with true

74. Replace #ifTrue: with #ifFalse:

75. Replace #ifTrue:ifFalse: receiver with false

77. Is not new … - History Begins in 1971, R. Lipton, “Fault Diagnosis of Computer Programs” Generally accepted in 1978, R. Lipton et al, “Hints on test data selection: Help for the practicing programmer”

78. Why is not widely used? Maturity Problem: Because Testing is not widely used YET! (Although it is increasing)

79. Why is not widely used? Integration Problem: Inability to successfully integrate it into the software development process (TDD plays a key role now)

80. Why is not widely used? Technical Problem: It is a Brute Force technique!

83. Number of Mutants: 1005

84. Time to create a mutant/compile/link/run: 10 secs. each aprox.?

85. Total time:

86. 6693300 seconds

89. Number of Mutants: 1005

90. Time to create the metamutant/compile/link: 2 minutes?

91. Time to run the tests per mutant: 1 sec

92. Total time:

93. 1125 seconds

95. Run all the test for each mutant

97. Mutate only covered methods

98. Run all methods per mutant

100. Create a mutant for each method

102. Create a mutant for only covered methods

104. Mutate Covered, Run Covering: 36 seconds

105. Result:

106. 545 Killed

107. 6 Terminated

109. MuTalk OptimizationsTerminated Mutants Try to kill the Mutant! The killer has to be “Terminated” The Test Suite

113. Filter Operators on View

115. a = b ifTrue: [ … ]

116. a = b ifFalse: [] is equivalent to a ~= b ifTrue: []

117. Suggest tests using not killed mutants

119. Why does it work? “In practice, if the software contains a fault, there will usually be a set of mutants that can only be killed by a test case that also detects that fault” Geist et al, “Estimation and enhancement of real-time software reliability through mutation analysis”, 1992

120. More Statistics…

122. But:

123. Mutants on not covered methods will survive

124. It provides better insight than coverage

126. MuTalk - MutationTestingfor Smalltalk Hernán Wilkinson UBA - 10Pines hernan.wilkinson@gmail.com Nicolás Chillo UBA nchillo@gmail.com Gabriel Brunstein UBA gaboto@gmail.com