Technical Presentation for the Plenary 2 meeting of the SmartDelta meeting.

1. AI For Software Engineering:
Two Industrial Experience Reports
Prof. Serge Demeyer
Technical Presentation, SmartDelta Project Meeting
Antalya, Turkey
November 2022

2. AI For Software Engineering © Serge Demeyer
Agenda
2

3. AI For Software Engineering © Serge Demeyer
Test Amplification
3
Program
Under
Test
Input Expected output
(mutation)
coverage
+coverage
Extra Input
Extra Input
+Extra Input +Extra output

4. AI For Software Engineering © Serge Demeyer
Example - testDeposit
4
1 def testDeposit (self) :
2 self.b.set_owner(’Iwena Kroka’)
3 self.b.deposit(10)
4 self.assertEqual(self.b.get_balance(), 10)
5 self.b.deposit(100)
6 self.b.deposit(100)
7 self.assertEqual(self.b.get_balance() , 210)
I
n
p
u
t
Expected
output

5. AI For Software Engineering © Serge Demeyer
Example - testDeposit_amplified (1/2)
5
1 def testDeposit_amplified (self) :
2 self.b.set_owner(’Iwena Kroka’)
3 self.b.deposit(10)
4 self.assertEqual(self.b.
5 get_transactions(), [10])
6 self.assertFalse(self.b.is_empty () )
7 self.assertEqual(self.b.owner, ’Iwena Kroka’)
8 self.assertEqual(self.b.get_balance(), 10)
…
Assertion
Am
plification

6. AI For Software Engineering © Serge Demeyer
Example - testDeposit_amplified (2/2)
6
1 def testDeposit_amplified (self) :
2 self.b.set_owner(’Iwena Kroka’)
3 self.b.deposit(10)
4 self.assertEqual(self.b.
5 get_transactions(), [10])
6 self.assertFalse(self.b.is_empty () )
7 self.assertEqual(self.b.owner, ’Iwena Kroka’)
8 self.assertEqual(self.b.get_balance(), 10)
9 with self.assertRaises(Exception):
10 self.b.deposit(−56313)
11 self.b.deposit(100)
12 self.b.set_owner(’Guido van Rossum’)
13 self.assertEqual(self.b.
14 get_transactions(), [10])
…
Input Am
plification

7. AI For Software Engineering © Serge Demeyer
AmPyfier Engineering
7
AmPyfier: Test amplification in Python
Journal of Software Evolution and Process 2022
https://doi.org/10.1002/smr.2490
Arti
fi
cial
Intelligence
Inside

8. 11 pull requests
8 merged
3 pending

9. Data
The planner
AI

11. AI For Software Engineering © Serge Demeyer
Python Project
11
20%
80%
Project Test Suite
• Start-up established in 2020
• Fast-growing
• more customers
• more investors
• Single variant
• AI engine is vital
(Unit) Tests

12. AI For Software Engineering © Serge Demeyer
Data Driven Test Amplification
12
Amplification operators must manipulate .csv files
import ampyfier
@ampyfier.amplify_csv ( "CSVPATH" )
def test_something(amp_file="something.csv"):
...

13. 360 minutes

14. AI For Software Engineering © Serge Demeyer
Experimental Set-Up
14
Daily
Ampyfier executes
tests on main
branch
Weekly
Ampyfier
generates new
tests

15. AI For Software Engineering © Serge Demeyer
Initial Amplification
15
Improvement for 13/25 files
• Mutation Score: 20.29%
• Code Coverage: 41.39%

16. AI For Software Engineering © Serge Demeyer
Continuous Amplification
16
Improvement for all
• Mutation score: 7/8 ≈ 87%
• Code coverage: 5/8 ≈ 63%
• Both: 4/8 ≈ 50%

17. AI For Software Engineering © Serge Demeyer
Agenda
17

18. Description text Mining
Stack Traces Link to source code
Product/Component
Speci
fi
c vocabulary
Suggestions?
Arti
fi
cial
Intelligence
Inside

19. AI For Software Engineering © Serge Demeyer
Results
19
Question Cases Precision Recall
Who should fix this bug? Eclipse, Firefox, gcc
eclipse: 57%
firefox: 64%
gcc: 6%
—
How long will it take to
fix this bug? (*)
JBoss
depends on the component
many similar reports: off by one hour
few similar reports: off by 7 hours
What is the severity of
this bug? (**)
Mozilla, Eclipse, Gnome
mozilla,
eclipse:67% - 73%
gnome:
75%-82%
mozilla,
eclipse:50% - 75%
gnome:
68%-84%
(*) In CSMR2012 Proceedings (**) In CSMR2011; MSR 2010 Proceedings
Arti
fi
cial
Intelligence
Inside

20. AI For Software Engineering © Serge Demeyer
Results
20
Question Cases Precision Recall
Who should fix this bug? Eclipse, Firefox, gcc
eclipse: 57%
firefox: 64%
gcc: 6%
—
How long will it take to
fix this bug? (*)
JBoss
depends on the component
many similar reports: off by one hour
few similar reports: off by 7 hours
What is the severity of
this bug? (**)
Mozilla, Eclipse, Gnome
mozilla,
eclipse:67% - 73%
gnome:
75%-82%
mozilla,
eclipse:50% - 75%
gnome:
68%-84%
Who should
fi
x this bug? Eclipse, Firefox, gcc
eclipse: 57%
fi
refox: 64%
gcc: 6%
—
Irrelevant for
Practitioners
Arti
fi
cial
Intelligence
Inside
(*) In CSMR2012 Proceedings (**) In CSMR2011; MSR 2010 Proceedings

21. AI For Software Engineering © Serge Demeyer
Story Points (Planning Poker)
21
1/2 1 2 3 5 8 13 20 40 100 ♾
Public Domain

22. Human
MMRE: 0.48
(*) Mean
Magnitude of
Relative Error
Learning Curve

23. AI For Software Engineering © Serge Demeyer
Arti
fi
cial
Intelligence
Inside
“in vivo” Validation
23
Explainable!

24. AI For Software Engineering © Serge Demeyer
Agenda
24

25. AI For Software Engineering © Serge Demeyer
Q&A support
25

26. In CHI2016 Proceedings
Arti
fi
cial
Intelligence
Inside

27. AI For Software Engineering © Serge Demeyer
Conclusion
27