Mobile Performance Testing

1. © 2014 SOASTA. All rights reserved. June 8, 2015 1CONFIDENTIAL – Not for Distribution
6
5
4
3
2
1

2. © 2014 SOASTA. All rights reserved. June 8, 2015 2CONFIDENTIAL – Not for Distribution
Lee Barnes – Founder and CTO of Utopia Solutions
Tom Chavez– SOASTA Sr. Product Evangelist

3. © 2014 SOASTA. All rights reserved. June 8, 2015 3CONFIDENTIAL – Not for Distribution
o Device/OS diversity
o Network variability
o Device performance
o User location and volume

4. © Utopia Solutions
Best Practices for Performance Testing Mobile Apps
Lee Barnes, CTO
Utopia Solutions

5. © Utopia Solutions
Apology
I don’t believe in “best practices”…
I do believe in guidelines and better
approaches for a given situation

6. © Utopia Solutions6
Key Questions
How is mobile different?
How can I adapt?
Why is performance of mobile
important?

7. © Utopia Solutions7
Why Should I Care?
M-Commerce
66%
OF
SHOPPERSABANDO
NTRANSACTIO
NS
5 OUT OF 10
BECAUSE OF
POORPERFORMAN
CE
1DELA
Y
SECOND
7% DROP
IN
CONVERSIO
NS
equals a
2013 Mobile Commerce Insights
study conducted by Jumio

8. © Utopia Solutions8
Still Not Convinced?
2013-2014 World Quality
Report

9. © Utopia Solutions9
Evolution of Performance Testing
1990 2000 2010
2-Tier
Client Server Web Mobile
Challenges
State of the
Practice
Testing Large
Loads
Complex /
Integrated
Systems
Proprietary
Protocols
Skilled
Resources
In-Lab
Testing
Tool
Maturation
Open Source
Tools
Specialized
Resources
Cloud-based
Testing

10. © Utopia Solutions
Mobile Challenges & Adaptations

11. © Utopia Solutions11
Mobile Challenges – User Access
Native App Mobile Site
Full / Responsive
Site
Your users have options…

12. © Utopia Solutions12
Assess the User and Load Profile
1. Determine the user profile
• Business processes
• Native app, mobile site, full site
• Preferred browser
• Network conditions
• Geographic location
2. Determine the load profile
• Volume of users by business
process and location
• Volume of users by access type
• Frequency / throughput
Incorporate constant feedback from application logs
and other monitoring solutions

13. © Utopia Solutions13
Mobile Challenges - Network
Mobile User Connected User
3G 4G InATunnelG
Network bandwidth and quality affect both end user
experience and system load

14. © Utopia Solutions14
Assess Impact of Network Variability
1. Assess baseline performance
(single user)
Observe UX with real devices
behind network virtualization
solutions
2. Assess performance under
load
Put system under virtual user load
Observe UX with real devices behind
network virtualization solutions
NV
Virtual Users
NV
Real
Devices

15. © Utopia Solutions15
Network Virtualization Tools
Single User (Device and/or
Emulator)
• Apple Network Link Conditioner
• Android Emulator
• Charles Proxy
Multiple Virtual Users
• Charles Proxy

16. © Utopia Solutions16
Mobile Challenges – User Location and Volume
A high volume of global users is difficult to reproduce in a lab…

17. © Utopia Solutions17
Utilize Cloud-based Testing Solutions
• Achieve production level loads
• Test entire infrastructure (vs.
behind the firewall)
• Distribute load geographically
• Assess user impact on real
devices

18. © Utopia Solutions18
Mobile Challenges – Device Performance
Measuring System / Network Metrics is Not Enough…
Memory
CPU
Battery Drain
Graphics
I / O

19. © Utopia Solutions19
Measuring Device Performance
Critical device metrics:
• CPU
• Memory
• Battery
• Storage
Mobile development platforms
• Android DDMS
• iOS Instruments
Mobile test automation tools
• TouchTest (SOASTA)
Mobile lab / cloud solutions
• deviceConnect (Mobile Labs)
• Remote TestKit (NTT Resonant)
Capture metrics while exercising app
Include appropriate devices /
configurations
Incorporate automation for consistency
Device Performance
Measurement Solutions

20. © Utopia Solutions20
Summary
Key takeaways…
• Mobile users are not the same as
connected users
• Performance is more than backend /
network
• Push performance analysis to the left
in the development cycle

21. © 2014 SOASTA. All rights reserved. June 8, 2015 21CONFIDENTIAL – Not for Distribution
SOASTA PRODUCT DEMO

22. © 2014 SOASTA. All rights reserved. June 8, 2015 22CONFIDENTIAL – Not for Distribution
o CI and Mobile Lab Scorecard
• 2 days
• Evaluate People, Process, Products, CI
• TouchTest and CI POC with customer tests
• Same mobile automation engineers that have implemented
mobile test automation for Fortune 5 and Fortune 10 retailers

23. © 2014 SOASTA. All rights reserved. June 8, 2015 23CONFIDENTIAL – Not for Distribution
UTOPIA
o Lee Barnes
o Founder and CTO
o Utopia Solutions, Inc.
o Email: lee.barnes@utopiasolutions.com
o Twitter: twitter.com/USI_LeeBarnes
o LinkedIn: linkedin.com/in/leebarnes
o Blog: www.RaiseYourMQ.com
SOASTA
o Tom Chavez
o Sr. Product Evangelist
o SOASTA
o Email: tchavez@soasta.com
o Twitter: @TomChavez
o LinkedIn: linkedin.com/in/tomchavez