Diese Präsentation wurde erfolgreich gemeldet.
Wir verwenden Ihre LinkedIn Profilangaben und Informationen zu Ihren Aktivitäten, um Anzeigen zu personalisieren und Ihnen relevantere Inhalte anzuzeigen. Sie können Ihre Anzeigeneinstellungen jederzeit ändern.

Test Automation Research... Is That Really Needed in 2018?

94 Aufrufe

Veröffentlicht am

We run a k€ 21,752 EU project to push the test automation research front. This talk motivates why this is (tax) money well spent and presents some research highlights: 1) test result visualization, 2) mutation testing, and 3) AI-assisted bug assignment.

Veröffentlicht in: Technologie
  • Als Erste(r) kommentieren

  • Gehören Sie zu den Ersten, denen das gefällt!

Test Automation Research... Is That Really Needed in 2018?

  1. 1. Research Institutes of Sweden TEST AUTOMATION RESEARCH… IS THAT REALLY NEEDED IN 2018? @mrksbrg mrksbrg.com Markus Borg Swedish Institute of Computer Science
  2. 2. Intrinsic human fascination with automation Research helps to automate: 1) better 2) more 3) sustainably (CC Flickr: dalbera)
  3. 3. Development engineer, ABB, Malmö, Sweden 2007-2010 ▪ Editor and compiler development ▪ Safety-critical systems PhD student, Lund University, Sweden 2010-2015 ▪ Machine learning for software engineering ▪ Bug reports and traceability Senior researcher, RISE AB, Lund, Sweden 2015- ▪ Software engineering for machine learning ▪ Software testing and V&V 3 Who is Markus Borg?
  4. 4. 4 ▪ TESTOMAT project ▪ Test visualization ▪ Mutation testing ▪ AI-assisted bug report assignment Agenda
  5. 5. TESTOMAT project 5
  6. 6. Problem Statement ▪ Contemporary dilemma. Modern software teams must optimize for both: ▪ bug free software ▪ ease of change ▪ Ever-faster release cycles => more automation 6 Project Goal ▪ Help software teams to increase the development speed without sacrificing quality ▪ Advance the state-of-the-art in test automation
  7. 7. 7 Plan Design Develop Maintain Execute Analyze Take action Three years 34 partners € 21,752,000
  8. 8. TRL 0 TRL 1 TRL 2 TRL 3 TRL 7 TRL 8 TRL 9 WP4 Test Priorities WP6 Test Automation Maturity WP5 Automated Testing for Quality Standards WP3 Test Effectiveness From technology validated in research labs To technology demonstrated in relevant environment TRL 4 TRL 5 TRL 6 8
  9. 9. Test visualization 9
  10. 10. 10
  11. 11. Large amounts of test results Simple result matrices hide information 11 The backside… Wikimedia Commons, Brukar:Bep Creative Commons CC-BY 2.5
  12. 12. 12 Solution Approach Visual analytics (Illinois Applied Research Institute)
  13. 13. ▪ Why game engine? ▪ Interaction out-of-the-box ▪ Why Unity? ▪ Fairly simple ▪ Scales well ▪ Very popular ▪ Unity??? ▪ Cross-platform game engine and IDE ▪ Drag-and-drop 2D and 3D scenes ▪ Scripting in C# 13
  14. 14. 14
  15. 15. 16 Plan Design Develop Maintain Execute Analyze Take action
  16. 16. Mutation testing 17
  17. 17. Do you trust your test cases? 18 Ali Parsai (Staff Sgt. Ryan Callaghan, US Air Force)
  18. 18. High Quality Test Suite Fewer Bugs How do you know that your test suite is good?
  19. 19. Test Suite Software 0
  20. 20. Fault Injection Mutant 1
  21. 21. SurvivedKilled 2
  22. 22. 3 Mutation Operator
  23. 23. a-ba+b a<<ba>>b a!=ba==b a<=ba<b !aa B::bA::b a(b,c)a(b) a|ba&b Very many test executions to kill mutants!
  24. 24. 25 Solution Approach Toward mutation testing in the cloud Sten Vercammen ▪ Goal ▪ Make mutation testing fast enough to fit during nightly build ▪ Approach ▪ Distribute the work ▪ Investigate bottlenecks and recommend optimizations
  25. 25. 26 Lessons learned from proof-of-concept tool ▪ Most steps are independent ▪ Speed-ups of 12x-13x with 16 workers ▪ Good chance to finish during nightly builds
  26. 26. 27 Plan Design Develop Maintain Execute Analyze Take action
  27. 27. AI-assisted bug assignment 28
  28. 28. 29 Dr. Leif Jonsson
  29. 29. 30
  30. 30. Bug tracker
  31. 31. 32 Model of bug report assignment Bug tossing!
  32. 32. ▪ Goal ▪ Useful tool deployable with minimum configuration effort ▪ Approach ▪ Bug reports = textual data + basic metadata ▪ Train machine learning classifiers on historical bug reports ▪ Combine them using state-of-the-art ensemble learning 33 Solution Approach AI-assisted bug report assignment
  33. 33. 34 Experimental setup Company A Company B Machine learning 4 x Pre-processing & feature selection 50,000+ bug reports
  34. 34. ▪ In line with human activity – But instantaneous! 35 Results
  35. 35. ▪ Productification of solution in internal tool ▪ Simplified solution without ensemble ▪ Deployed in bug tracker for large project ▪ Presents instantaneous recommendation of responsible team ▪ Accuracy 8% lower than manual work 36 Prototype deployed at Ericsson
  36. 36. 37 Bug tracker Machine Learning
  37. 37. 38 Bug tracker Machine Learning
  38. 38. 39 Plan Design Develop Maintain Execute Analyze Take action
  39. 39. Research on test automation = Good use of tax money 40
  40. 40. Intrinsic human fascination with automation Research helps to automate: 1) better 2) more 3) sustainably (CC Flickr: dalbera)
  41. 41. Research Institutes of Sweden Swedish Institute of Computer Science markus.borg@ri.se mrksbrg.com @mrksbrg © Musée des arts et métiers-CNAM Research => 1) better 2) more 3) sustainable test automation

×