Sarah Lovely who is a Performance Tester at Cognizant Technology Solutions along with Raja RajKaliappan who is a part of Non Functional Testing CoE at Cognizant Technology Solutions took a Session on "Performance testing of Chatbot" at Global Testing Retreat #ATAGTR2019 Please refer our following post for session details: https://atablogs.agiletestingalliance.org/2019/12/04/global-testing-retreat-atagtr2019-welcomes-sarah-lovely-as-our-esteemed-speaker/ https://atablogs.agiletestingalliance.org/2019/12/04/global-testing-retreat-atagtr2019-welcomes-raja-rajkaliappan-as-our-esteemed-speaker/

1. #ATAGTR2019
Performance Testing of Amazon Lex Chatbot &
Alexa enabled Smart Speakers
Raja Kaliappan (Raja.RajaKaliappan@cognizant.com) & Sarah Lovely S (SarahLovely.S@cognizant.com),
Cognizant
14th-15th Dec 2019

2. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
Market Predictions for 2020
Chatbots & Smart Speakers
Key Market Players Benefits
Customer service interactions
will be powered by Chatbots85%
Businesses expected to have
some Chatbot automation
implemented
80%
US households will have Voice
Assistant Smart Speakers75%
138 Million Voice Assistant
Smart Speakers will be in
use with an audience of
258 million
138MM
Amazon Lex
IBM Watson
Microsoft LUIS
Google DialogFlow
Amazon Echo devices
enabled by Alexa
Google Home
24/7 Service
Faster Service
Business Automation
Enhanced UX
Cost reduction
Scalability

3. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
Amazon Lex - Chatbot service
Amazon Lex
 An AWS service that enables building
conversational interfaces using voice and text
 Provides deep functionality and flexibility of Natural
Language Understanding (NLU) and Automatic
Speech Recognition (ASR) to build highly
engaging user experiences
 A fully managed service that scales automatically
with no need of infrastructure management
 Pay-as-you-use model and charged based on
number of requests
FEATURES
Natural Language Understanding
Automatic Speech Recognition
Seamless Deployment
Auto Scaling
Cost Effective
Interoperability with AWS services
Text & Voice Support
Ease of Development
KEY BENEFITS
Simple Bot development through
conversation flow definition
Auto scalable platform eliminating
infrastructure overheads
No upfront cost and cost for idle time

4. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
Amazon LEX / Alexa – Interaction Model
Wake word Launch Invocation Name Utterance
Alexa, tell Domino Pizza, I need a pizza
Lex identifies the user intent as Order Pizza
Slot
Sure, what size pizza do you want?
Lex prompts to elicit the Slot Size
Prompt
Slot Value
I want a medium sized Pizza
Lex captures the Slot value Medium
I have placed your order and you will be
receiving your pizza shortly
Lex fulfills the Intent by placing the order

5. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
AWS Lambda – Code hooks for Lex
AWS Lambda
Serverless platform letting one run code without
provisioning (or) managing servers
Execution of Lambda functions (Your code) is
event driven and can be triggered from other
AWS services like Lex, S3 events, DynamoDB,
API Gateway etc.
Automatically scales and launches as many
copies of the code as and when triggered and
executes them in parallel
Cost effective with Pay-as-you-use model and
you are charged only for number of requests and
the duration of the execution
LEX Chatbot Architecture
Lambda Role in a Lex Chatbot
 Initialization & Validation code hooks
(E.g. Validation of data inputs from the user – Claim Number format validation)
 Fulfilment code hooks
(E.g. Calls to On-premise Claims application for getting Claims Status)

6. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
Lex Chatbot Performance – Key Considerations
Critical Intents
 Identify business critical Intents, test for maximum number of
possible utterances so AI feature of the Bot is put to test under load
Peak Workload
 Maximum concurrent connections to Lex and maximum request rate
for the critical intents
End User
Response Time
 Define & validate SLA for End user response time
Lex Latency  Validate & set baseline for Lex Latency
Lambda
Execution Time
 Monitor and improve Lambda Execution time with code and/or
memory optimization
Lambda Memory
Configuration
 Configure Optimal memory for Lambda function considering
performance & cost

7. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
Lex Chatbot Performance Test Approach
Performance Testing & Monitoring Approach for a Lex Chatbot
Performance &
Scalability Assurance
High Availability Optimized Cloud Cost
Best End User
Customer Experience
Key Benefits
Lex API Tests Lex Monitoring Lambda Monitoring Enterprise Services
Monitoring
Load test Lex API for
different throughput &
Lambda memory
configurations and optimize
both performance & cost
Leverage CloudWatch
dashboard to monitor
Lex metrics like Lex
Latency, Lex
Throughput etc.
Monitor built-in Lambda metrics
in CloudWatch and build &
monitor custom metrics like
memory consumption by
extracting CloudWatch logs
Leverage Enterprise APM
tools like Dynatrace,
AppDynamics for monitoring
on premise Infrastructure &
Application
E2E Tests
Simulate load from UI to
capture end user
performance for different
throughputs & validate
against SLA

8. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
 Chatbot Initial Connection for first few initial requests observed to
take higher time
 End User & Lex API Response time affected by user proximity /
Request origin
 Lex Latency observed to be consistent for different test user loads
 Lambda Cold Starts causing higher response time for initial few
utterances
 Lambda Execution Time improves with higher memory configuration
but at cost of higher price
 Lambda Overbilled Duration can be a factor in considering for cost
reduction
Lex Chatbot Performance - Observations

9. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
Lex Chatbot Performance – Best Practices
Balance both Lambda Performance &
Cost by right memory configuration
Maintain proximity between User base
location, Bot orchestration & AWS
resources to avoid network latency
Best Coding practices to optimize
Lambda Execution Time and optimize
Overbilled duration
Implement plugins to keep Lambda
containers warm at right intervals to
avoid Cold starts

10. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
Alexa Architecture
Request Voice Stream
Response Voice Stream
Request Text
Response Text
Alexa Voice Service Custom Alexa Skill
+ Alexa Skill Kit
AWS Lambda
Enterprise Services
Other AWS Services
Alexa enabled
device
User
API test for Alexa Skill API
Performance
E2E Response Time ?
Speech Recognition
Natural Language Processing
Interaction model
definition
Logic Execution
Backend Calls

11. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
Web Speech API
Web Speech API
Converts input text in the web page into
speech using device’s speech synthesis
system and plays it out of a device’s
speaker
Speech Synthesis (Text-to-Speech)
Receives speech through a device’s
microphone and converts into text through
either device’s or server based recognition
system & outputs the text in the web page
Speech Recognition (Speech-to-Text)

12. #ATAGTR2019
As a author of this presentation I/we own the copyright and confirm the originality of the content. I/we allow Agile testing alliance to use the content for social media marketing, publishing it on ATA Blog or ATA social medial
channels(Provided due credit is given to me/us)
Alexa End User Performance Evaluation
 Automates real world voice
based interactions and assesses
end-to-end performance of Smart
Speakers like Amazon Echo
(Alexa) & Google Home devices for
a single user
 Automated reports with Response
time metrics and SLA
validation highlighting outlying
utterances
 Identification of performance
issues of smart speakers at
the end user level
 Elimination of human effort
required for voice based
testing with automation of test
execution & reporting
 Need for automation of voice
based testing for Smart speakers
 Used in unison with load testing
of Alexa Skill API / Lex API to
capture end user response time
of Smart speakers
Features BenefitsWhere can it be used?
In-browser Speech Synthesis for
pre-defined user utterances
In-browser Recognition of voice
responses from Smart Speakers
Powered by
Web Speech API
Alexa End User Performance Evaluation Solution using Web Speech API
How it works

13. Thank You