by Shubhradeep Nandi, Head of Digital, MSys Technologies at STeP-IN SUMMIT 2018 15th International Conference on Software Testing on August 31, 2018 at Taj, MG Road, Bengaluru

1. STeP-IN SUMMIT 2018 - Machine Learning
AI driven test automation
in the AI first world
Shubhradeep Nandi

2. AI First World
❖ Technology projects predominantly are now Datascience Projects
➡ Guided by data
➡ once live requires no manual intervention
2
Images belong to their respective owners and this is just for illustration purposes only
❖ What needs to be done for adoption
❖ Exhaustive testing on the claims of the product(QA team)
❖ Explicability of the underlying model(Both DEV & QA team)

3. ❖ Machine Learning
❖ Deep Learning but Why ?
❖ Classification in the AI world
3
Refreshing a few basic concepts
Images belong to their respective owners and this is just for illustration purposes only

4. ❖ Before - We were
testing an
Application.
❖ Now - We are
still going to test
an Application.
4
What did not change?
Images belong to their respective owners and this is just for illustration purposes only
What got changed?
❖ Before - Rule based AI(Automation with
pure test/train) for testing a rule based
workflow driven application.
❖ Now - Self Learning AI for testing a
combination of
‣ Deterministic -The workflows that are
rule driven in the app
‣ Non-Deterministic -The workflow that
are driven by AI with learning elements
The ‘Shift’ in Testing Paradigm

5. Deterministic vs Non- Deterministic
5
Images belong to their respective owners and this is just for illustration purposes only
What????

6. Insurance driven by AI
6
Open up the App
Upload your supporting docs
Pay the Premium
Customer Onboarding
Auto Verification
Deterministic rule based
Non Deterministic driven by AI
Images belong to their respective owners and this is just for illustration purposes only
Register by filling up the form

7. Insurance driven by AI
7
Claim Settlement
Upload image of damage
Real time validation and estimation using AI
Claims are settled
Open up the App
Deterministic rule based
Non Deterministic driven by AI
Images belong to their respective owners and this is just for illustration purposes only

8. The AI mindset
❖ What is AI?
❖ Is it all Math?
❖ Is it all Algorithm?
❖ Is it all Programming?
❖ Does it all depends on the tool/team that builds it?
8

9. Creating the holistic test framework in the AI first World
❖ Identify and Understand the scope of the Application
❖ Segregate the Deterministic Workflow from the Non
Deterministic Workflow
❖ AI for testing the Deterministic workflow of the App
❖ AI for testing the Non Deterministic workflow of the
App
9

10. 5 blocks to understand the scope of application
What is the input for the application?
1. Define the possible inputs captured
2. Define the order of capture
10
Understand the application flow
1. The business requirement
2. The behavioural workflow
Segregate the Non-Deterministic Portion
1. Learning models involved
2.Learning model behaviour matrix
What is the hierarchy?
1. Identify the primary/secondary workflow
2. Identify the learning workflow
Quantify the learning property
1. Data Properties
1. Size of Data
2. Noise Level
2. Degree of Automation - Automatic/Semi-Automatic
3. Supervision - Supervised/UnSupervised/Reinforced
4. Time(Online/offline or Lazy/Eager)

11. Segregating the Deterministic Workflow from non-
Deterministic Workflow
11
Deterministic Non Deterministic
Images belong to their respective owners and this is just for illustration purposes only

12. AI for testing the Deterministic workflow(non AI) of the App
12
1 2 3
1. Expose deterministic steps of ML
2. The application Blueprint is created
3. The Cognitive Script generation
Automation with cognitive approach
Images belong to their respective owners and this is just for illustration purposes only

13. AI for testing the Deterministic workflow(non AI) of the App
❖ Recorder and Meta Modelling to automate scripts generation
❖ Elastically scale functional, load and Performance Testing using deep forecast models
❖ Self Healing Tests with Deep Learning
❖ Analytical reports and Visualisation for explanations
13
Automation with Deep Learning approach
Images belong to their respective owners and this is just for illustration purposes only

14. AI for testing the Non Deterministic workflow(AI) of the App
14
Images belong to their respective owners and this is just for illustration purposes only

15. Object Classification and Detection - The most sought after AI
usecase
❖ There is an enormous rise of autonomous vehicles, smart
video surveillance, facial detection and various entity
identification applications
15
Images belong to their respective owners and this is just for illustration purposes only

16. 16
❖ Some use cases are very critical -
❖ Outcome should be highly accurate
❖ Objects has to be detected, classified, and, delivered in fraction of a second
Object Classification and Detection - The most sought after AI
usecase
Images belong to their respective owners and this is just for illustration purposes only

17. 17
❖ Deep learning framework CNN(Convolutional Neural Network)
has achieved the state of start in Object Classification and Detection
Object Classification and Detection - The most sought after AI
usecase
Images belong to their respective owners and this is just for illustration purposes only

18. How to build a AI framework to test this Deep Learning usecase ?
18

19. Approach to build the framework
1.Checking the CNN robustness using Perturbations
using Generative Neural Networks - Fellow et al.
2.Neural Network correctness with Linear Programming
or SMT Solvers - Katz et al.
19
Images belong to their respective owners and this is just for illustration purposes only

20. Approach to build the framework
3. Systematic approach with Synthetically generated
Datasets
20
Scalable
Realistic instead of
perturbations
Images belong to their respective owners and this is just for illustration purposes only

21. Modules of the Framework
21
Generator Sampler Visualizer
Generate realistic
Images of objects
Provide modification
points to the Image
generator
Sampled Modifications
against Metrics of
Interest

22. Generator
❖ Modification functions are used to represent a subset of feature space.
❖ Low dimensional modification allow us to test Convolutional Neural
Networks on a compact domain
22
Mathematical Relationship
f : X —> Y
X’ ⊆ X
Generator (y : M —> X’)
Every Modification(m) m ∈ M
y(m) ∈ X’
X —> Feature Space
Y—> Output
M —> Modification Space
y —> Modification Function
m —> Individual Modifications
Images belong to their respective owners and this is just for illustration purposes only

23. Sampler
❖ Identify a low discrepancy sequence methodology to
competently produce sample sequences that provide
high coverage of the abstract space.
➡ D(K,X) = abs(#(K)/m - vol(K))
❖ Capitalise on the Active Learning capability to reduce
process expense.
➡ Gaussian Progression for non parametric regression
23
D—> Dimension space
K —> subset of Dimension Space,
m —> modification as tupple
vol —> Volume

24. Visualizer
❖ Intersection over Union and CNN Confidence Score -
A standard approach to measure accuracy for Object
Detection and Classification.
24
Images belong to their respective owners and this is just for illustration purposes only

25. Connecting the 3 Modules
function ANALYZERCNN
repeat
p ← sampler(P)
x ← generator(p)
y ← f(x)
D.add(m, x, y)
until condition(D)
visualizer(D)
end function
25

26. Where can I apply this?
26
Images belong to their respective owners and this is just for illustration purposes only

27. Few of the Many…
❖ Healthcare - Radiology, Cardiology, Dermatology…
❖ Insurance - Claims, Customer on-boarding…
❖ Life Sciences - Drug Discovery, Pharmacovigilance…
❖ Manufacturing - Industrial Vision, Quality Inspection…
27

28. Thank You
28
Email :- shunandi@gmail.com or shubhradeepn@msystechnologies.com
Images belong to their respective owners and this is just for illustration purposes only