AI improves software testing by Kari Kakkonen at TQS

2
• ROLES
• Knowit Solutions Oy, Director of Training and Competences,
Lead Consultant, Trainer and Coach
• Secretary of ISTQB Executive Committee
• Treasurer of Finnish Software Testing Board (FiSTB)
• Children’s and testing author at Dragons Out Oy
• ACHIEVEMENTS
• Influencing testing since 1996
• Ranked in 100 most influential IT persons in Finland (Tivi
magazine)
• Great number of presentations in Finnish and international
conferences
• TestausOSY/FAST founding member.
• Co-author of Agile Testing Foundations book
• Regular blogger in Tivi-magazine
• EDUCATION
• ISTQB Expert Level Test Management & Advanced Full &
Agile Tester certified
• DASA DevOps, Scrum Master and SAFe certified
• SPICE provisionary assessor certified
• M.Sc.(Eng), Helsinki University of Technology (present Aalto
University), Otaniemi, Espoo
• Marketing studies, University of Wisconsin-Madison, the
USA.
BUSINESS DOMAINS
• Wide spread of business domain knowledge
• Embedded, Industry, Public,
• Training, Telecom, Commerce,
• Insurance, Banking, Pension
SERVICES
• ISTQB Advanced, Foundation and Agile Testing +
Knowit Quality Professional
• DASA DevOps
• Quality & Test process and organization
development, Metrics
• Agile testing, Scrum, Kanban, Lean
• Leadership
• Test automation, Mobile, Cloud, DevOps, AI
• Quality, Cost, Benefits.
Kari Kakkonen twitter.com/kkakkonen
linkedin.com/in/karikakkonen/
Dragonsout.com
Copyright Knowit Solutions Oy 2020 #TQS2020

2400
Employees, 150 in
Finland
In 5
countries
Sweden, Norway,
Denmark, Finland and
Germany
315 million
Turnover
Nasdaq
OMX Mid Cap
30 million
Profit, margin 9,6 %
Copyright Knowit Solutions Oy 2020 #TQS2020 3

• Artificial Intelligence (AI) in short
• Case of NASA code quality metrics
using AI
• Other AI and testing developments
Agenda

Drivers of artifical intelligence
1. Models everywhere.
2. Amount of data is exploding.
3. Manual analytics is slow and costly.
• Interest to automate analytics and
modeling is growing.
• Development speed of analytics
algorithms is accelerating.
• Algorithms, that automatically process
data to information, can produce data
and better models independently.
Figure: Amount of data in world. Forbes, 27 Nov. 2018.

Quality management as top use-case for AI
Source: AI software platform adoption survey, IDC, February 2019 [Percent of respondents: N=505]

• When talking about AI, it is
important to ask if we are
talking about modelling
humans or how to work in
ideal way.
• Machine learning is a sub-
category of AI, which
considers algorithms that
enable AI to learn ideal way to
work.
Artificial intelligence == machine learning?
Figure: Example of concepts related to AI.

Three methods of machine learning
Machine learning methods
Supervised Learning
• The aim is to learn
mapping from input to
output.
• Output is known by
supervisor.
Unsupervised Learning
• The aim is to find
”something interesting”.
• Output is not known
(known also as density
estimation in statistics).
Reinforcement Learning
• Seemingly good
solutions are
encouraged to
develop further.
AI literature review[1]
1 Reagan, A., et al., The emotional arcs of stories are dominated by six basic shapes, EPJ Data Science (2016).

No one tool for all
Performance
Amount of data, time, and computing power
• Linear discriminant
analysis
• Logistic regression
• Medium-sized ANNs
• Genetic algorithms
• Extreme learning
machines
• Convolution neural
networks
• Generative neural
networks
• Deep belief networks
• Recurrent neural
networks
• Simple neural networks
9

Planning
descriptions
Code Unit testing
Integration testing
System testing
Acceptance
testing
Definition
descriptions
Requirements
• Static methods
• Reviews
• Dynamic methods
• Test runs
Remember there are different kinds of testing

AI in software testing
• In static testing, several machine learning
algorithms can be used to measure code
quality.[1] NASA’s code review.
• In dynamic testing, model-based regression
tests can be replaced by algorithms that
generate test flows on-the-fly.[2] Chaos
monkey of Netflix.
• In maintenance testing, machine learning can
be used to plan more realistic statistical
models of how application is actually used.
Figure: Margaret Hamilton standing next
to source code of guidance system of
Apollo spacecraft.
www.nasa.gov/
11

Toolbox for AI

Code quality metrics: NASA case

• Detecting:
• abnormal behaviour
• new kind of customers
• etc.
Anomaly detection
Figure: Example of time series
14

• Several code quality metrics are currently
being used. The most popular ones have
been introduced in 1970.[1,2]
1 T.J. McCabe, A Complexity Measure, IEEE Transactions on Software Engineering (1976) SE-2, 4, 308.
2 M.H. Halstead, Elements of Software Science, Elsevier (1977).
3 N.E. Fenton, S.L. Pfleeger, Software Metrics: A Rigorous & Practical Approach, International Thompson Press (1997).
4 S.J. Sayyad, T.J. Menzies, The PROMISE Repository of Software Engineering Databases (2005).
• Goal is to provide feedback:
1. for developers about code quality.
2. for testers about needed test coverage.
3. for management about the development
project
• Although traditional metrics are extensively
used, benefit of using them has been
questioned.[3]
0
50
100
150
200
250
300
350
400
450
0 20 40 60 80 100 120
Numberoflines.
Cyclomatic complexity
Scattering of defects in NASA CM1 project
OK Defected
Figure: NASA Metrics Data Program, CM1 Project[4]
In real applications defected modules do not separate from
non-defected by standard metrics.
AI code quality metrics

AI review
Requirement
management
Code quality
analysis
Version
control
Test
management
Defect
management

Developer
Product
Owner
Scrum team
Management AI review

• Goal is to understand technical quality and test coverage of modules.
Also, goal is to guide and focus testing:
• Focusing technical debt reduction
• Found defects per module
• Guide for test coverage in automating system level testing.
Requires that there is link from components (version control) and tests (test management) to feature descriptions and from reported defects to modules (observation control)
Module Developer Benchmark
(order.numb.)
Relative
risk
Number of
reported
bugs
Module_1 Jaakko 50 58 4
Module_2 Jaakko 4 5 1
Module_3 Mikko 180 95 7
SCRUM TEAM
18

• Goal is to provide overview:
• How fast the development is advancing
• Maturity for release
• Quality and technical debt of the software
• Defect probabilities
• Risks related to modifications of the
software.
Interpretation:
• Module 5 has high defect probability and is
related to five most important features of
the application. This results into high risk in
release at this moment.
• Defect probability of module 10 has fallen
to acceptable level.
Start-up
phase
Development
phase
Maturing
phase
MANAGEMENT
19

1
2
𝑤 2
+ 𝐶 ෍
𝑖=1
𝑁
ξ𝑖
1-𝑦𝑖 𝑤, 𝑓(𝑥𝑖) + 𝑏 ≤ ξ𝑖
Minimize
Subject to with all values of i.
Quality metric 1
Qualitymetric2
• Machine learning approach was tested for NASA’s CM1
software repository[1].
• Support vector type of machine learning algorithm was
developed for this demonstration.
• This task can be solved by supervised machine learning
methods because both input (quality metrics) and output
(defects) is known.
• Goal is to find such hyperplane in multidimensional data
space that separate defected modules from non-
defected.
• Mathematically, problem can be described with algorithm:
AI-guided software testing, demonstration

0,0
0,2
0,4
0,6
0,8
Tested on real data against NASA CM1 project[1]
Performance(F)
Training set: 1595
Test set: 400
𝐹 =
2 ∗ 𝑎𝑐𝑐𝑢𝑟𝑎𝑐𝑦 ∗ 𝑟𝑒𝑐𝑎𝑙𝑙
𝑎𝑐𝑐𝑢𝑟𝑎𝑐𝑦 + 𝑟𝑒𝑐𝑎𝑙𝑙
AI-guided software testing, demonstration

Stages of execution
4. DEPLOYMENT AND UTILIZATION OF THE ALGORITHM
Monitoring and analysing results
2. DEFINING QUALITY METRICS
Defining quality and defect metrics
Building the metrics
3. DEFINING HOW RESULTS ARE PRESENTED
Defining who will use results and how
Building data visualization
1. DEFINING DATA SOURCES
From where data is collected
How will the data be collected

Other interesting developments in AI
and testing

• Enhanced automation
• Test automation
• Robotic Process Automation -> Intelligent automation
• AI-powered test automation
• Enhanced test selection
• Risk analysis
• Test case selection
• Code analysis
General direction at businesses
on AI and testing

• Start from screenshots
• Detect layout issues
• Cross-browser testing for differences
• Ignore invisible rendering, size, position differences
• Case: Applitools
• https://applitools.com / Visual AI
AI-powered comparison of visual elements

• AI-based test generation
• Checking functional differences
• Checking visual differences
• Automatically learning the test
automation
• For: regression testing after changes
• Case: retest
• Unfortunately a victim of Covid-19
• https://retest.de/ai-based-test-
generation/
AI with regression testing

• “World’s first”
• Go automatically through mobile apps
• Use general test cases
• Reinforcement learning
• Case: test.ai
• https://www.test.ai/
AI-powered mobile test automation platform

• Default tests to web sites
• Teach your own tests
• Running on containers in virtual
machines
• Use for regression testing, cross-
browser testing
• Case: Mabl
• https://www.mabl.com/
Auto-healing automated testing of web

• Build test through interface backed by ML
• Plain English
• New life for “record-playback”
• ML algorithms maintain the tests
• Case: Functionize
• https://www.functionize.com/
Intelligent testing

• Support test automation engine data selection with AI
• Can be used for testing with large quantities of data
• Can be used automating production as Robotic Process Automation
(RPA)
• Cases:
• Robot Framework test automation framework
• Aito AI tool
• https://robotframework.org/
• https://aito.ai/
• https://aito.ai/blog/machine-learning-in-robot-framework/
Automating tests or production with lots of
repetitive data

• Go through functionalities
• Identify risky areas
• Focus testing
• Case: Eggplant
• https://www.eggplantsoftwar
e.com/
Predict quality issues

• Analyze risk in commits to the code
• Select the tests that test the risky areas
• Reduce number of test cases needed to run
• Case: Appsurify
• https://appsurify.com/
Select most suitable test cases

• Power of open source
developers
• Machine learning
algorithms learn from the
community automatically
• Analyse code and
propose improvements
• Case: DeepCode
• https://www.deepcode.ai/
Code analysis using AI

• Trainings and certifications
• https://www.alliance4qualification.info/a4q-ai-and-software-testing
• https://www.brightest.org/en/certifications/AiU-Certified-Tester-in-Artificial-
Intelligence/
• Associations
• https://www.aitesting.org/
• www.istqb.org
• Standards
• ISO
• Focus on AI https://jtc1info.org/technology/artificial-intelligence/
• Under development: Testing of AI-based systems
https://www.iso.org/standard/79016.html?browse=tc
• IEEE
• https://ieeexplore.ieee.org/document/8705808
Promoting and teaching AI and testing

• https://www.pnsqc.org/wp-content/uploads/2018/09/38-Santiago-AI-
Driven-Test-Generation.pdf
• https://ieeexplore.ieee.org/document/8705808
Research

• AI-enabled testing is already a reality
• Many companies are enhancing their solutions with AI
• New companies are set up around AI
• AI can provide simplicity to complex software development projects.
• With AI, testing activities can be focused on high risk areas.
• With AI, test automation becomes more autonomous
• Testers are freed to created new tests
Conclusion

Thank You!
Twitter
@kkakkonen
Email
kari.kakkonen@knowit.fi

AI improves software testing by Kari Kakkonen at TQS

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (12)

Ähnlich wie AI improves software testing by Kari Kakkonen at TQS

Ähnlich wie AI improves software testing by Kari Kakkonen at TQS (20)

Mehr von Kari Kakkonen

Mehr von Kari Kakkonen (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

AI improves software testing by Kari Kakkonen at TQS