We co-hosted a webinar with Neotys to shed some lights on - How to overcome the challenges in mobile app performance and functional testing - How to gain granular and actionable insights to measure and improve your app user experience - Best practices to get the mobile readiness for 2017 Holiday Shopping Season - A brief demo of the integration between Neoload and Bitbar Testing

1. The Best of Both Worlds - Combining Performance
and Functional Mobile App Testing

2. HouseKeeping rules

3. Presenters
Speaker
Henrik Rexed
Performance Engineer
Neotys
Speaker
Lingkai Shao
Marketing Analyst
Bitbar
Speaker
Dave Horner
Director of Sales
Bitbar

4. Neotys Continuous Performance Testing Solution
System Testing
Integrated Component
Testing/System-wide Testing
ReleaseComponent Testing
Continuous Integration &
Component Testing
Proactive Production
Monitoring End-User
Experience

5. “Continuously ship quality apps with a high velocity delivery pipeline”
Bitbar - Mobile DevOps

6. Dev Ops
Continuous
Release &
Deployment
Collaborative
Development
Continuous
Testing
Continuous
Monitoring
Continuous
Integration
Continuous
Feedback
DevOps is no longer a
unicorn’s discipline.
TestDev Ops
Shift Left Shift Right
DevOps

7. According to Adobe 2016 Online Shopping Data, mobile drove 55% of visits to retail websites on Black Friday (45%
coming from smartphones, 10% from tablets) and accounted for 36% of sales (25% from smartphones, 11% from
tablets) *
Why Functional and Performance Testing for Holiday Season
*=http://news.adobe.com/press-release/marketing-cloud/media-alert-adobe-data-shows-black-friday-breaks-online-sales-record-3

8. Performance matters
53% of visits to mobile sites are
abandoned after 3 seconds
according to research from Google
According to Google doubleClick faster websites has 70% longer
average session lengths, 35% lower bounce rates and 25% higher ad
viewability than their slower counterparts.
Etam reduces its average page load time from 1.2s to 500ms
and increased conversions by 20%
Walmart saw up to a 2% increase in conversions for every 1 second of
improvement in load time.
Every 100ms improvementalso resulted in up to a 1%
increase in revenue.

9. Top Failures During Peak Days

10. Concrete study

11. When Revenue is Generated

12. ● Preparation: typically 10 weeks ahead of peak days
● Functional test suite including
Stage One: Functional & Compatibility

13. Do you know the behavior of your app/web users?
Defining the right user flows

14. Stage One: Functional & Compatibility

15. Stage One: Functional & Compatibility

16. Stage Two: Performance & Load

17. Standard performance testing workflow
Build the
performance
strategy
Create the testing
scenarios
Prepare
Execute
Analyse
Tuning

18. Performance design is the key

19. Mobile Application Architecture Landscape
Native app
Hybrid app
Web app
Web app
Web Model
Mobile Model
Rest Interface
Business logic
Data Access logic
XML
Legacy
Relational
Presentation Business Data
Third
party

20. Mobile Performance Testing Touchpoints
BACK-END
EXECUTION PLATFORM
RIA/FRAMEWORKS
DATABAS
E
NETWORK
3G, 4G / LTE, Wi-Fi
VIDEO
HTTP Adaptive
StreamingThroughput
Latency
Packet loss
SERVER
INFRASTRUCTU
RE
OS
CLIENT
APP
DEVICE
iPhone 4, 5, 6
Samsung
Blackberry
…
(7,000 different device types are used to access
Facebook)
Screen Size
CPU
Memory
Native
Browser-Based
Hybrid

21. Impact of Mobile Network

22. The most important network constraints are:
• Latency
• Bandwidth
At a certain point, increasing the bandwidth will only have a minor effect.
Latency & Bandwidth . . .

23. Packet Loss Consequences
• Packet Retransmission
• End user experience degraded because packet loss increases jitter
• Errors could occur if packet loss is in pure data transmission
Packet Loss is Inevitable with Mobile
• Mobility: Quickly switching to the next cell tower may result in temporary signal loss due to congestion
on the current tower
Mobile Protocols Try to Reduce Packet Loss Rate
Packet Loss . . .

24. Lots of users
• Mobile applications often have large user bases
• High growth in popularity can create the need for highly scalable load generation infrastructure
Geographical distributed load
• Users are accessing apps from locations around the globe
• Distance and location can greatly affect performance
Mix of Web & Mobile Users
• Many mobile apps have a corresponding web app with a shared backend
• High load on the web app can affect performance on mobile
High volume, Geographically Distributed

25. Don’t forget the User Experience…..
What are is really
being measured?

26. … Applying the Bitbar Testing Integration…
00:00:00 00:00:30 00:01:00 00:01:30
Load
NeoLoad
Application
Server
App Error Rate
Virtual Users Load
User
Experience

27. RUM (Passive) vs Synthetic (Active)
Stage Three: Proactive Production Monitoring

28. Why Synthetic Monitoring?
● Alarm and Alert
● Better MTTR! (Mean Time To Resolution)
● Find and fix issues before impact to users
● Baselining and Benchmarking (Pre-production Staging & Post Production)
● Prepare for peak traffic and new markets
● Monitor Complex Transactions
● Measure and Adhere to SLAs
● Hold 3rd Party Vendors Accountable
● Test from end user perspective
● Competitive Benchmarking
Stage Three: Proactive Production Monitoring

29. Understand Performance of Complex User Transactions and Hold 3rd Party Services Accountable (SLAs)
Benefits of Synthetic Monitoring

30. Faster Mean Time to Repair… Identify Issues Before Your Users Do
Benefits of Synthetic Monitoring

31. Understand Performance In New Markets, Optimize Content Delivery, Benchmark Pre & Post Change
Benefits of Synthetic Monitoring

32. Competitive
Benchmarking:
Benefits of Synthetic Monitoring

33. Purpose Built Synthetic Monitoring Solution

34. Global Points of Presence

35. Use Your Existing Functional Test Scripts For Monitoring

36. Solution:
Stage Three: Proactive Production Monitoring

37. DEMO
Demonstration

38. Questions?
A
Q