Interactive Session on "Use of AI and ML in Performance Testing" by Adolf Patel Performance Test Architect Cognizant at #ATAGTR2021. #ATAGTR2021 was the 6th Edition of Global Testing Retreat. The video recording of the session is now available on the following link: https://www.youtube.com/watch?v=ajyPSmmswpM To know more about #ATAGTR2021, please visit:https://gtr.agiletestingalliance.org/

1. #ATAGTR2021
Use of AI and ML in Performance
Testing
Adolf Patel

2.  Overview
 Artificial Intelligence & Machine Learning
 Aspect of AI in testing
 How AI and ML can help in performance testing
 AI and ML based strategy for performance testing
 How AI and ML are changing performance testing process
 Detecting performance anomalies using AI & ML
 Benefits of AI-ML in performance testing
 Q&A
Agenda

3. Overview
In today's world of digital era where performance testing is a must to do activity before production
rollout, Artificial intelligence(AI) and Machine Learning(ML) is a game changer.
Despite of many automation tools available in the market which reduces human effort in
developing the scripts, executing the test scenarios and analysing the result, still, there is good
deal of manual/human efforts required when it comes to evaluate the results. Use of AI in
performance testing will make tasks like scripting, monitoring highly impactful and help to
get real time results very quickly and more accurately.
AI is crowding the technological landscape at snowballing speed. Appl’s Sri, Microsoft’s Cortana or Google’s
Assistance are some of the common AI powered tools we hear in everyday life. How about something similar for
performance testing like, you say, “Hey Sri, please send me the performance metrics of XYZ application on my
email” and you get a nice report appear in your mailbox or you say, “Hey Sri, please execute an endurance test for 4
hours for XYZ application and send me the report to my email” and your test environment spin up and execute the
test and send the report to your email. It sounds very impressive, isn’t it? Can it be done or even possible now? AI is
here, and it's bringing a whole new dimension to the performance testing.

4. AI is a system with increasingly self-learning capabilities, that can
complement human reasoning and activities by understanding the
environment, solving human problems, and perform human tasks.
In simple, Artificial intelligence is the simulation of human intelligence
processes by machines, especially computer systems.
Machine learning is a method of data analysis that automates analytical
model building using historical data as input to predict new output values. ML
is a subset of Artificial Intelligence. The machines learn from the history to
produce reliable results.
• Supervised Learning - Supervised learning algorithms analyze the
training datasets data and produce an inferred function. Example, Credit
card fraud detection.
Artificial Intelligence & Machine Learning
• Unsupervised Learning - Unsupervised learning algorithms analyze and learn the un-clustered data
and produce an inferred function. The algorithm itself finds the patterns in the data. Example,
Facebook friend request suggestion mechanism.
• Reinforcement Learning - Reinforcement learning is defined by characterizing a learning problem
and not by characterizing learning methods. Example, bots

5. Aspect of AI in testing
In the world of Agile, IT Managers want the code to build quickly, test quickly
before the production and deploy the code to production. If any errors comes
during testing, fix the issue and repeat the testing lifecycle quickly. More than
80% of the teams follow Agile methodology for software development
and 70% of the release happens weekly or before weekly basis.
Functional issues can be found and tested before production but
finding a performance issue or predicting the performance issue or
pattern in the production is bit difficult. So, can we use AI solution
to make things faster and smarter? AI brings the ability to
anticipate problems appearing in a system can prevent any
unforeseen issues in app development
Not only will AI increase the reliability of predicting system
breakdown in general operations, but test engineers will also be able
to use AI to create test strategies focused on recognizing dangers
most likely to appear.

6. How AI and ML can help in performance testing
• Quick and Better Data Analysis – Reduce error and improve data usages.
• Enhanced user experience – Conventional load testing involves server side
performance testing using some protocol like HTTP, This does not give the real user
experience performance. AI can be useful to load test the real browser based load
testing which give the better user experience performance. Tool like Loadninja, a
testing platform on the cloud to load test web applications with real browsers at
scale.
SMART
Identify performance trends with more granularity – Use machine
learning(ML) to dig deep into the data and highlight key performance
bottleneck for QA and Dev.
Create SMART SLA’s – A SMART SLA helps service providers to set
stakeholder expectation according to current as opposed to the past
system behavior. It also enable test engineers to create performance
tests scenarios according to the latest performance data available rather
than depending on the past data every time they perform the test.
Prevention is better than cure – AI help predict the issues before it occur thus giving time to
engineers to fix before it pops up.

7. AI and ML based strategy for performance testing
Planning Test Design Test Execution
Analysis &
Reporting
Production -
Live
• NFR Collection
• SLA Definition
• Script Creation • Workload Modelling • Smart Analysis
• Issue finding
• Tunning
Recommendations
• Live Monitoring
Issue, Fixes & Retest
Application Updates/Maintenance
Collect
Analyse
Learn
Predict
AL/ML Assisted
Components
AI-powered application monitoring and
scripting tools can detect issues in real
time, pinpoint the source of the
problems and automatically launch the
proper remediation processes — often
before end-users even recognize that
there’s been any kind of problem.
Application/
Environmen
t

8. How AI and ML are changing performance testing process
Make realistic test data – Setting up test data for a performance test scripts are not so easy. Many times, you need
to do it manually in csv files, sometimes you capture dynamic data from response packets and use for next course of
action in your scripts. An ML assisted tools or simple ML program can help building the vast range of data in no times.
Ensure system is resilient - Many of us are familiar with Chaos Monkey, the Netflix tool that intentionally pulls down
production instances, given that every system should be redundant. The objective is to test if a system is pulled
down, there should be no impact to system performance. By monitoring performance when tests are running,
management can determine when redundancy is lacking. Picking systems and randomly pulling them down is
arguably a type of AI. By noticing what is failing and what is working, Chaos Monkey can produce more complex
strategies for breaking things, a sort of true AI that can improve the customer experience.
Provide Realtime insight into a system’s performance - ML solutions can analyze and interpret millions of data in
no time, providing near to real time insight into systems performance. They can provide patterns and make
predictions to solve performance issues faster and more accurately than human, hence improving the efficiency.
Predicting performance in production – Can we predict how the application will perform under
certain load in production? The answer is may be. The simplest application of ML is to use data
sources to predict trends. Trends help leaders tell if a situation is under control, getting better, or
getting worse and how fast it's improving or worsening.

9. As enterprise software platforms expand in complexity and usability, performance anomalies have
become a serious threat that can result in huge loss in cost and reputation of an enterprise. Machine
learning solutions can analyze and interpret thousands of statistics per second, providing real-time
insight into a system’s performance.
There are several ways machine learning can be utilized to detect performance anomalies in
performance. It keeps baselining the real time behavior and comparing against the new normal for
any anomalies. Here are few of the most popular methods.
Detecting performance anomalies using AI & ML
Poor User
experienc
e
Failed
Load
Balancing
Blurred
Video
Streamin
g
Decrease
d
Throughp
ut
Server
Crash
Freezing
Hanging
Increased
Latency
Delayed
Response
Time
Density-Based – Assuming that normal data points occur around a dense neighborhood and, therefore, anomalies
are far always, this method of detection is based on the K-Nearest neighbors' algorithm, or alternatively the local
outlier factor.
Clustering-based- This method is one of the most popular concepts for unsupervised learning. It assumes that data
points that are similar usually belong to similar groups. Here, a K-means algorithm creates ‘k’ similar clusters of data
points, with instances that fall outside the clusters being marked as potential anomalies.
Support vector machine-based – Support vector machine is generally associated with supervised learning. Here the
algorithm learns a soft boundary to cluster the normal data and then identifies instances that fall outside the learned
region.

10. Benefits of AI-ML in performance testing
 Codeless automation using natural language processing(NLP)
 Testing environment developed using ML helps in self healing and corrections
 Test flows can be automated and can be tested using data
 Performance test modelling process provide real time workload during test execution
 SMART SLA preparation and real time workload modelling
 Adapting the performance rules over time and inventing new rules for automated
remediation
 Extrapolating the resulting data to human experience of performance
 Increased used of AI assisted APM tools