1.
1

2.
• Progress of AI and Robotics
• What’s the need for Artificial Intelligence?
• What will happen at singularity?
• Some AI Concepts
High level Intro to AI
• Is it an Intelligent Activity?
• Are we testing at the heights of
Augmented General Intelligence?
• AI in Testing - Is it augmented or Artificial
/ Is anything artificial about it?
• How will AI evolve Testing?
• Some Examples of AI Testing
AI in Testing
Agenda

3.
- ANDREW NG
Founder of Coursera, Stanford Adjunct Professor
Ex. Chief AI Scientist of BAIDU

4.
4
Current Understanding of AI -

5.
Source : https://blogs.nvidia.com/blog/2016/07/29/whats-difference-artificial-intelligence-machine-learning-deep-learning-ai/
A Good Explanation of Progress of AI
E a r l y A I
Basic Turing Test Style
Use of Memory and Knowledge
Post John McCarthy’s Conceptualization
Basic Robotics and Degrees of Freedom
D e e p L e a r n i n g
Rapid Infrastructure Growth
Advanced Algorithms
Big Data Explosion
Quantum Computing
M a c h i n e L e a r n i n g
Algorithms Centric
Statistics Driven
Supervised and Unsupervised Learning

6.
Perhaps the greatest
Computer Scientist ever
predicting on Machine
Intelligence
We have clearly passed
the TURING TEST
We are seeing Leaps and
Bounds in advances of
Technology!
Let’s hear!

7.
Neal Creative | click & Learn moreNeal Creative ©
Singularity or transcendence is around the
corner – Aided by AI

8.
https://youtu.be/zatL4uFRpC0
Fast Learning – Download and Fly an helicopter
Can AI Take us to this stage?
How about Fast Testing – Hey – Can I test this brand new “thing” in 2 minutes?

9.
From a Leader in AI – AI or an Algorithm Writing itself

10.
Top Human World Champions Royally defeated by AI!
2011
2016
1996/97
IBM’s
Deep Blue
IBM
Watson
Google
Deepmind
AlphaGo won 60–0 rounds on two public Go websites
including 3 wins against World Go champion Ke Jie.

11.
But... AI is not without controversies though!
Facebook Researchers shut down an AI
engine at the Facebook AI Research Lab
(FAIR), discovering that the AI created
its own unique language undecipherable
by humans - Simultaneous glimpse of
both the awesome and horrifying
potential of AI
Elon Musk - “AI isotentially more
Dangerous than Nukes”
sets up a $1 billion (£770M) OpenAI.org to try
and promote safe development of AI
Vladimir Putin -
“Whoever masters AI will rule the world!”
ISAAC ASIMOV’s Laws of Robotics
Law 1: A tool must not be unsafe to use.
Law 2: A tool must perform its function
efficiently unless this would harm the user. The
safety of the user is paramount.
Law 3: A tool must remain intact during its use
unless its destruction is required for its use or for
safety.

12.
Japan is using AI and Robots in Multiple ways

13.
https://quickdraw.withgoogle.com/ , https://www.autodraw.com/ & https://g.co/aiexperiments
Google is using Deep Learning to make Simple things –
Better!

14.
https://oxism.com/thing-translator/
Take it to next level
How can you easily integrate globally. Language
will no longer be a barrier.

15.
AI Startups are taking it to next Level – in all areas
Source: https://www.cbinsights.com/research-ai-100
Bots Automobile
Computer
Vision
Core /
Functional AI
Commerce IIoT/IOT
Healthcare Fintech Robotics
Analytics
Cyber
Security
Sales &
Marketing

16.
Let’s get to
TESTING!

17.
“ Let’s get to TESTING
• How is AI helping
Testing?
• How can we test
better with AI?
• How can we test AI
systems Better?
17

18.
“Be a yardstick of
QUALITY. Some
people aren't used
to an environment
where EXCELLENCE
is expected”
18
Steve Jobs

19.
Business Agility - Some Statistics
19
Google - refactors code by
50% each month*
Netflix - 5 Billion+ API
Calls per Day (and
increasing daily)
~75% of Corporates to
have bi-modal IT
~63% all projects are not
aligned to Business
Strategy
~79% organizations using
CI/CD/DevOps practices in
one form or the other
52% of Fortune 500
companies have
disappeared from the list
& Average S&P500 span
reduces from 61 Years to
17 Years in 60 years
In 2020, 100 million
consumers will shop via
augmented reality
By 2020, 30% web
browsing will be done
without a screen
by 2022 - $1 Trillion a year
to be saved through IoT
Source: Gartner, Inc. Top Strategic Predictions for 2017 and Beyond: Surviving the Storm Winds of Digital Disruption, 14-Oct-2016
* - Google runs on ~2 Billion LOC Source: CA Workshop on Modern Software Factory
Source: CB Insights
* AR Market $143 Billion by 2020 - HW/SW/Apps/Consulting & SI

20.
Is the Testing Industry ready for testing the
following innovations?
21

21.
Tip of the Iceberg seen in 2016
2016 A Year in Review – Software Failures
22Source: Tricentis Software Fails 2016 Report - https://www.tricentis.com/wp-content/uploads/2017/01/20161231SoftwareFails2016.pdf
Over 4.4 Billion
people got affected by
a Software Fail (Up
from 4.3 Billion in
2015) > 50% Global
Population
$1,062,106,142,949
- Assets Affected (Up
from $4.2 Billion in
2015)
315 years, 6 months,
2 weeks, 6 days, 16
hours, & 26 minutes -
Accumulated time-
lost due to Bugs
2.66 Billion Mobile
Phones impacted
with Malware
12% Year on Year
Increase in impactful
Software Bugs
British Airways lost
$20 Billion (3%) in
Market Cap within a
few days after a failed
software upgrade
More Than 21
Million Automobile
recalls as a result of
Glitches / Bugs
$5.7 Billion Impact
in Failed Government
Software Projects due
to Bugs
2.2 Billion people live on less than $2 a day

22.
One School of Thought on Testing – By Tricentis
Source: TRICENTIS webinar on Future of Testing
WhereAIcanhelp
Legacy
Firms
Bi-model Firms
Technology Leaders

23.
Test
Coverage
GAP
Years
Months
Weeks
Days
Hours
Seconds
TestingDuration
Challenge of Complexity, Less time and more Tests
Where we are
in time
Testing Complexity
Time

24.
26
Resources
Effort
CostsCoverage
#of Test
Cases
When you have less time overall

25.
Some Algorithms making Machine Think!
Source: https://futurism.com/predicting-2017-the-rise-of-synthetic-intelligence/ - Some of the artificial intelligence (AI) algorithms currently helping machines think. Credit: CIO Journal/Narrative Science

26.
Approaches used for AI, Machine Learning and Deep Learning
Reinforcement Learning
• Passive Reinforcement
Regression Algorithms
• Linear Regression
• Gaussian Process
Supervised Learning
• Neural Networks
Unsupervised Learning
• Independent Component
Analysis
• Principle Component
Analysis
Natural Language
Understanding
• Morphological, , semantic,
syntactic , Discourse
analysis
Natural Language
Generation
• Deep planning
• Syntactic generation
Clustering Algorithms
• K-Means Clustering
• KPCA – Kernel Analysis
Statistical Algorithms
• Support Vector Machines
• K-Nearest Neighbor
• Native Bayes Classifier
• Maximum Entropy Classifier
Pattern Recognition
• Statistical , Syntactic
approach
• Template Matching
• Neural Networks
Other Techniques
• Spanning Trees and Graphs
• Neural Network – Multi-
Level Perceptron's
Other Techniques
• Labeling
• Hidden Markov Model
• Maximum Entropy MM
Other Techniques
• Conditional Random Fields
• Parsing Algorithms

27.
29
Some methods to build AI Data Models for Testing
1 2 3 4 5

28.
What are the feasibilities
with AI Driven Testing?
30
Automated Defect
Detection
Automated
Exploratory
Testing
Test Coverage
Heat map
Self Healing
Automation
Predictive
Modeling
Self Adjusting
Regression
Pattern
Recognition
Risk & Coverage
Optimization
Diagnostic,
Prescriptive and
Predictive Analysis
Deep Learning
Root-Cause
Analysis
Sentiment Analysis

29.
31
AI Models Algorithms
Application Under
Test
Designer Developer Business UserTesterBots / Agents
AI Engine
Testing Outcomes
Test Cases
Production
Logs
Requirements
Defect Logs Source Code
Traceability
Matrix
Root Cause
Analysis
Test Data
Specifications
Functional
Logic
Sample AI Model for Testing
Historical & Real-time Data

30.
Let’s see an
example

31.
Example: Candy Crush Saga’s AI Strategy
https://www.youtube.com/watch?v=wHlD99vDy0s
• Use of AI engine for continuous Feedback Loop
• Use of BOTS to perform Testing
• Continuous Feedback Loop
• Deep Artificial Neural Network
• Use of Monte Carlo Tree Simulation
• Use of Advanced Automation by BOTS
• Hybrid Test team (150-200+ Testers) with unique skills
• Use of Data Scientists for Domain Knowledge, Fun (using
historic info and user behavior, Game Balancing)
• Regular Crash Testing, Performance Testing, Regression
Testing
• Regular Upgrade of AI Bot for Testing
v

32.
Recommendations
/ Summary

33.
AI Testing Recommendations
Summary