Testing Strategies to Deliver Consistent App Performance

www.neotys.com© 2015 Neotys. All Rights Reserved.
30th of June
Henrik Rexed Manjunath Premkumar
Technical Evangelist
Neotys
Performance Test Architect
Symphony Teleca- HARMAN
NO MORE GAMBLING
WITH
APPLICATION PERFORMANCE
DURING GRAND NATIONAL
&
SEASONAL SPIKES

Agenda
Case study
Recommended testing approach
Betting industry has to use latest technology to ensure good UX
Performance Matters
Know your Customer
Major Betting Operators

Meet the most connected customer
Retail Shops
FitBit
iWatch
Narrative
Camera
Samsung Gear
Smart TV
Phones and
Tablets
Gaming
Console
…and customers will be as connected in the next 2 years

What they expect from Betting Operators
“I WANT TO PLACE MY BETS ON ANY
GAME/SPORTS FROM ANYWHERE,
ANYTIME”
“I WANT ACCESS TO FUNDS”
“I WANT TO GET THE SAME GREAT
GAMING/SPORTS EXPERIENCE ACROSS
ALL MY DEVICES”
“I WANT YOU TO GIVE ME SUGGESTIONS
AND OFFERS THAT ARE RELEVANT TO ME”
“I WANT YOU TO SEND ME REAL-TIME
ALERTS”
“I WANT YOU TO ENSURE SECURITY OF MY
DATA”

Performance to End User
What is PERFORMANCE to End User?
“Hassle free application with rich contents and quick response time at finger tip for user to perform actions like Login, Signup, Bet
placement, check wallet balances, funds transfer etc.”
Why PERFORMANCE Matters?
User experience is an important factor , even minimal differences in the response time can have significant impact on the business
Amazon
+100 ms
1% drop in sales
Yahoo
+400 ms
5-9% drop in requests
Google
+500 ms
20% drop in requests
Bing
+2000 ms
4.3% drop in revenue/user
Shopzilla
-5000 ms
25% increase in requests
7-12% increase in revenue
50% reduction in hardware
Mozilla
-2200 ms
15.4% increase in
downloads
KEY MESSAGES

• Websites are becoming richer in design and content
• Good user experience is most desirable attribute
• Web User turnaround starts from more than 3 seconds of response time
• Better interface demands performance optimization at all application levels mainly client side
“80-90% of the end-user response time is spent on the client side.”
Steve Souders, Google’s Head Performance Engineer
80-90% Client Side 10-20% Server Side
97%
www.aol.com
95%
www.ebay.com
95%
www.facebook.com
98%
www.msn.com
98%
www.myspace.com
94%
en.wikipedia.org/wiki
97%
www.yahoo.com
98%
www.youtube.com
TIME SPENT ON THE CLIENT SIDE
Performance Matters Continued

How Brain Perceives Performance

Major Betting Operators

We forecast more customers in betting and Gaming domain based on our research across competing operators, this in turn requires Betting Operators
to have strong focus on enterprise scaling for Seasonal Sports and Festivals like Grand National to maximize its customer acquisitions.
• Interim Q1 -2014-
Mobile net revenue
accounted for 50%
of the online revenue
• Record new
customer
acquisition of
37.2%
Source: Ladbrokes
Reports
Mobile revenue up
162% and the
channel contributed
70% of the
Sportsbook revenues
• 65% increase in the
number of active
customers in
sustainable markets.
Source: BetFair
Reports
Mobile net revenue
accounted for 52%
of the online
revenue, with 73% of
active Sportsbook
customers
• Record new
customer
acquisition of
795,000 (up 35%)
Source: Paddy power
Reports
• 1.5 million
downloads from app
store
• Mobile to generate
40% gaming
revenue by mid –
2015
Source: WH Interim
Reports
The Competetion IS ‘ON’

6440
3974
9354
1337
6561
5337
6841
8672
44654526
1514
3916
835
3240
1949
2150
4929
2302
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
Ladbrokes William
Hill
Paddy
Power
Bet365 Coral Betfred Skybet Betfair TitanBet
Desktop Channel
Document Complete - First
View (milliseconds)
Document Complete - Repeat
View (milliseconds)
WPT Report : Dated 20th-May -2015
Doc complete – user can perform action
on the page.
All DOM elements are downloaded.
Competitor Desktop APP Performance Analysis-
Homepage
• Bet365 appears to be having quickest response for first
time visitor with an average of 1.3 Seconds and repeat
user at 900 milliseconds.
• Paddy Power seems to have taken high response time for
DOM elements to be downloaded

Key performance testing challenges
Sustaining of Spike loads for short duration
Simulating realistic user behaviors across different
customer facing channels
Simulating real time sports updates (Bet in Play and
Non BIP sports) to client apps
Simulating realistic high volume transactional load
patterns - Bets placed, signups, Deposits,
Withdrawals for festive spirits like Grand National
Simulating tests on different clients apps-Desktop &
Mobile /bandwidth simulation – 3G, 4G, WIFI/ Regions –
UK and across the world
Understanding the underlying complex architecture and
identifying the key touch points (shared architecture) in
enterprise architecture
Maintaining high UX by introducing rich contents like
Personalization, stylesheets, Capturing Analytics ,
Complex JavaScript's and Interactive chat services.
Unavailability of performance environment like to like
with production environment

Architecture are becoming very complex

Web pages are bigger and more complex
2.3 objects per page
Average size 14k
75 objects per page
Average size 498k
83 objects
Average size 684k
1995 2010 2012

Latest Technologies used by Betting Operators
FRONT END
BACKEND
MIDDLETIER
BACKEND
CACHING STREAMING/PUSH
HTML 5,
CSS3,Sass, Jquery,
Flash, Node.js,
Angular.js, Flex,
Ajax
Java, .Net, Flash,
Javascript, Ruby,
Python, PHP,
Coffee script
Mongo DB,
Coherence,
Memcache,
Couchbase, CDN
HDS, HLS, Smooth
Streaming and RTMP
Websockets/Push-
Kaazing, Comet, Redis,
HTML5 Server Sent
events , Lightstreamer,
ICEfaces
Tibco, IBM
Websphere,
Talend, UtlraESB,
Apache Camel
Informix, Oracle,
DB2, IMS, MySQL

Early Involvement
Any early involvement would be beneficial
• Even if only asking a few key questions
• Don't wait until everything gets in place
Unfortunately, you often get involved in
the project at a later stage
• Next sections are still fully applicable

Let ’s take the time to understand the application
Performance is driven by:
• The end users
• Third party system
• …etc.
We need to ask the right questions:
• How the users works on the system
• What are their habits
• When ? and how often
• Are we going to expand in different geos?
• Is the marketing plan to market/promote
• the application? If yes what is the type audience?
• …etc.

Using the right methodology
Performance testing an
existing application
• Ask the support of functional
architect, HR..etc
• Understand the behavior of the
application with the help of the
logs
• Migration project : Don’t under
estimate the history of the
application
Performance testing a new
application/service
• Involve the project leader , functional
architect
• Try to understand the purpose and
relation with the business plan
Application is like a castle
• You want to enter the castle to load
what’s inside
• User pause would be the key
User’s habits are different
• Include relevant behavior in your tests
to properly stress your components
• Involve the business to identify the
ratio of users finalizing each user
journey

Grand national Performance Testing For a Betting
Operator
Business Context
One of UK’s leading Betting
and Gaming Company wanted
to enhance the customer
experience on web and mobile
applications with 99.9%
server uptime, quick
server response and
browser rendering time
Year’s highest turnover is expected on Grand National day for the Bookmaking Industry. Each
Bookmaking Industry wants to make the best of this event by making a good profit but on this
day most of Bookmaking Industry’s servers go down.
How it was done ?
CREATE THE
SCHEDULES
FIRM UP
NFRs
DESIGN
THE TESTS
EXECUTE
TESTS
REPORT &
RECOMME
ND
0 – FESTIVAL
PREPARATION KICKOFF
4
3
2
1

Google Analytics
Web Server Log
Analysis to capture
Realistic User Journey
Volumetric NFRs
on
Transactional Data
like No. of Bet
placement,
Deposits,
Withdrawals etc.
Test scripts were
designed with
bespoke
frameworks/tools
to mimic end
user’s action and
sports updates
Customized load
profile to mimic the
load distribution
pattern as realistic as
observed on GN day
Amazon cloud to
simulate the high
Enterprise user’s loads
and distribute the users
from different
geographies
Enterprise level
system monitoring,
coordination across
development teams,
Infrastructure team
and egaming support
teams
HOW DID WE GO ABOUT REALISTIC TESTS ?

TOOLSET FOR ENTERPRISE LOAD TESTS
WPT
• Client side tuning tool
• User experience on
browsers
• Private instance to test in
controlled environments
Correlsense
• Trace bottlenecks
• Capture real time metrics
• Enables quicker turn
around in fixing the
bottlenecks
Confluence
• NFRs
• Execution reports
• Documentation.
Neoload
• Mobile, Web, API, websocket,
bandwidth simulations, customized
reports etc.
• Identify bottlenecks in the early
stages
Amazon
Cloud
• To generate higher enterprise loads
• Mimic different geo
• Cost effective
• Cloud credit utilization model
Jira
• Defect management
• Issue resolution tracking
Wireshark
• Network monitoring tool
• Capture network-related
metrics
• Network Bottlenecks

KEY ENTERPRISE PERFORMANCE BOTTLENECKS FIXES
& DECISIONS
SL NO. STREAM BOTTLENECK FIX
1 Mobile High response to download the static contents
Code changes to implement progressive download and using
application cache and moving few static contents to CDN
2 DesktopMobile
Not able to scale to high volume of login and bet
placements
Doubled the APIs capacity of Backend App servers to support the
high volume loads
3 Mobile Webservers reaching capacity and causing high failures
Additional 7 no. of webservers were added to support high volume
loads
4 Network
Bandwidth utilization was reaching its capacity (95Mbps
out of 100 Mbps link) causing resulting in packet drops
and high response time for transactions
Network traffic was split to MPLS(100Mbps) and other on
VPN(100Mbps) to support high volume traffic
5 DesktopMobile
High response time for Login and Bet placement observed
at ESB layer
Addition Instances for Login and Bet placement , turn off logging
payload and bypass EMS servers
SL NO. STREAM CASE DECISION OUTCOME
1 DesktopMobile
Auto –top up module not scalable to higher volumes
and impacting others functionality due to shared
services
To Turn off Auto-top up functionality on GN day
To improvise user’s experience on the
banking transactions without any hassle on
GN day
2 DesktopMobile
High volume of Bet history and Transaction History
calls impacting server performances and other
modules hosted on the same servers due to shared
services
To Turn off Bet History and Transaction History
on peak hour of GN day
To reduce impact on user experience to
customers on GN day
3 DesktopMobile
Fraud rules check impacting the Login and Customer
Deposits transactions at higher volumes
To Turn off and Optimizing few Fraud rules
causing problems and retaining the key fraud
rules on GN day
To reduce impact to the customers carrying
out deposits on GN day

SOLUTION HIGHLIGHTS
Specific scenarios for AJAX calls, Push
requests were simulated as per production
website. Focus on improving Time to First
Byte (TTFB), Page Rendering and Full Load
for first view and repeat view
User load distribution across multiple
customer facing channels based on
historical stats from Google Analytics,
production web server logs. And real-time
sports updates were pushed to desktop
and mobile user interface to simulate
realistic behavior
SOLUTION SHOWCASE
Peak bets : 7000 bets/min. 1200
Deposits/Min, 700 signups/min and Peak
user load of 60,000+ concurrent users on
mobile and 25,000+ concurrent users on
Desktop. Tools used: Neoload (cloud +
on premise), Webpage test & bespoke
simulation tool
Maximised customer retention and
acquisition
300%
Response time
Improvement in
application
response time
Solution Showcase and Highlights

Key Factors for successful GN 2015 Readiness
Key Decisions
Driving Force – IT
and Business
Real time
monitoring
Risks/issues and
mitigation plan
Daily Meetings
Performance
Strategy
Collaboration
GN 2015

Next Steps
Email henrik.rexed@neotys.com and Manjunath.Premkumaraiah@Symphonyteleca.com
Visit Neotys Community at answers.neotys.com
Download NeoLoad Free Edition from www.neotys.com
Evaluation support at www.neotys.com/support/contact.html
Questions?
Visit www.neotys.com and www.symphonyteleca.com

40
GLOBAL OFFICES
8600
EMPLOYEES
23
COUNTRIES
Top 10
Mobile Companies
Top 5
Outsourced Product Development Companies
2012 Partner of the year
Windows Azure, Finalist
Excellence Award
Technology Agency of the Year 2015

Testing Strategies to Deliver Consistent App Performance

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (10)

Similar to Testing Strategies to Deliver Consistent App Performance

Similar to Testing Strategies to Deliver Consistent App Performance (20)

More from HARMAN Services

More from HARMAN Services (20)

Recently uploaded

Recently uploaded (20)

Testing Strategies to Deliver Consistent App Performance