2. XBOSoft
Dedicated to software quality improvement
Founded in 2006
We speed products to market with our expert:
• Software QA consulting
• Software testing
Global team with offices in San Francisco,Beijing
and Amsterdam
“Thorough, accurate and fast”
XBOSoft, Inc. All Rights Reserved. 2
3. House Rules
Participants other than the speakers are muted
Questions via the GoToWebinar control on the
right side of your screen or through Twitter
@XBOSoft
Questions may be asked throughout the webinar -
we’ll try to answer them at the end
You will receive info on recording after the webinar
XBOSoft, Inc. All Rights Reserved. 3
Webinar Hashtag: #MobileDeviceLab
4. Meet Our Speakers
Philip Lew
CEO and Founder, XBOSoft
• Relevant specialties and
passions
o Software quality process,
evaluation, measurement
and improvement
o Software quality in use /
UX design
o Mobile User Experience
and usability
o Cycling and travel
XBOSoft, Inc. All Rights Reserved. 4
Karla Smith
Marketing Manager, XBOSoft
• Emails you about our
industry hot topic
webinars
• Invites you to download
our latest trends in
software testing
whitepapers
XBOSoft Team
5. Guest Presenter
Costa Avradopoulos
XBOSoft, Inc. All Rights Reserved. 5
• Over 25 years’ experience in development, from
requirements, to delivery, and process engineering
• Roles in Development, QA, Automation, Product
Management, VP, and CTO. Management of
systems up to 15 million users in verticals such as
Telecommunications, Financial, Wireless,
Transportation, and Retail.
• Much of career dedicated to the mobility space, culminating with a breakthrough
patented mobile technology
• Frequent speaker. Writing book due to be published soon: Winning Mobile
Strategies – Bridging the Quality Gap
• Principal for Avracom. Advises organizations in the creation of winning mobile
strategies to develop best of breed applications.
• Certified Six Sigma and TPI
costa@avracom.com
www.avracom.com
6. Why We’re Here
XBOSoft, Inc. All Rights Reserved. 6
• Mobile applications – disruption in the market
and in the SDLC
• Mobile testing requires a different approach
• A good strategy starts with designing a good
mobile device lab
• With thousands of variations, traditional
approaches are insufficient
7. Agenda
XBOSoft, Inc. All Rights Reserved. 7
• Mobile Testing Challenges
• Defining Lab Requirements
• Building a World Class Device Lab
Out of Scope for Today:
• Discussion on specific tools
• Automation
8. XBOSoft, Inc. All Rights Reserved. 8
How Do You Define Mobile?
1 ABLE TO MOVE OR BE MOVED
FREELY OR EASILY. 2
OF OR RELATING TO CELLULAR PHONES,
HANDHELD COMPUTERS, AND SIMILAR
TECHNOLOGY.
Smartphones, Feature phones, tablets, laptops, ruggedized devices,
medical devices, wearables, kiosks, ATMs…
9. XBOSoft, Inc. All Rights Reserved. 9
World Quality Report 2014
KEY FINDINGS:
The QA function is adapting to business demands by streamlining and
centralizing its structure, to achieve efficiency and cost optimization
Significant investment in Test Environments may also be undermined
by a lack of specialist expertise
Testing’s late engagement in the application delivery lifecycle and its
reliance on basic IT metrics is possibly applying a negative brake on
this increasing maturity
Mobile testing has increased in importance as a key discipline within
the function, but lacks specialized methods, expertise and
environment
10. XBOSoft, Inc. All Rights Reserved. 10
World Quality Report 2014
KEY FINDINGS (Cont.):
Cloud adoption and cloud-based testing has slowed as organizations
continue to manage concerns regarding data security and
performance
A higher proportion of overall IT budget is being invested in testing
and focused on transformation projects
Agile development is now widely adopted but still gives rise to
problems for testing, particularly in relation to specific methodologies
and expertise
NOTE: Mobile Testing is a dedicated section in this report
To obtain a copy of this report, email
costa@avracom.com
11. Mobile Testing Areas of Focus - 2014
Source: World Quality Report, 6th edition
11
XBOSoft, Inc. All Rights Reserved.
12. The Cost of Poor Quality Escalates
65%OVER
of all apps have a
1 star rating or less
Top quality issues in last 12 months exceed $2
Billion in losses:
• Security: breach at top 10 major retailer ($570M+ loss)
• Performance/Outages: Top 3 social media company – 5 major
outages in 2014, and 2 outages in 2015
• Security: breach at top 10 major retailer ($500M+ loss)
• Functional: Top 3 bank, double payments issue
12
XBOSoft, Inc. All Rights Reserved.
13. Impact of Failure
Cost of Poor Quality is Accelerated with Mobile
Visibility is far greater, exposing your brand & reputation:
• Social Media makes failures exponentially more visible, with potential
to go viral
• App store is a public report card - you cannot hide or cover up poor
ratings
A plethora of choices for customers = competitive risks
• Near zero switching costs for customers
• 60% of users will abandon your app/site if does not load in 3 seconds
• Less than half will return to give a slow app/site a 2nd chance
13
XBOSoft, Inc. All Rights Reserved.
14. Mission
Critical
Low Criticality
Mobile Testing Challenges
Security
Usability
Performance
Networks
Fragmentation
Due to involvement of GSM/ CDMA carriers identification of threats,
risk and determining its business impact is crucial
Thorough understanding of Visual, Accessibility and certification
testing is a must
Due to the limited hardware architecture of mobile devices, success
and failure of application in market depends on performance of the
app
Testing to be performed on different networks like 3G, 4G, Wi-Fi and
Edge, etc
When you factor in different OS versions, unique hardware models,
and carrier versions, the permutations are in the thousands
Tools Specialized knowledge of tools specific to mobile testing
Lack of Awareness
There are many mobile tools and test approaches available in market,
but all are not mature
14
XBOSoft, Inc. All Rights Reserved.
15. Internal Cloud
Definitions of Cloud
What do we mean by:
PUBLIC CLOUD
A mobile lab hosted by the testing tool
provider. All cloud clients have access to
the devices in the lab. A specific tester
reserves the device he needs and uses it
when its available. Accessible through
the internet.
PRIVATE CLOUD
A mobile lab hosted by the testing tool
provider dedicated to a specific client.
The tester can reserve a device without
having to share the devices with other
clients. Accessible through the internet.
ON-PREMISE CLOUD
A secure mobile lab set up by the
client itself. All the devices are
connected to a server inside the
company network. The company has
complete control over what devices
are procured and who uses them.
External Cloud
15
XBOSoft, Inc. All Rights Reserved.
16. Shared Advantages
• Enables better collaboration between Dev/QA
• Both provide very robust lab/device
management, supporting provisioning and
maintenance
• Resource management. Both provide a
mechanism for scheduling/reserving devices
• Improve efficiency of test execution
• Enable continuous integration for DevOps
16
XBOSoft, Inc. All Rights Reserved.
17. Using an External Cloud
Pros:
• Scalability - easy access to hundreds of devices on demand
• No need to maintain devices (provider does it)
• Newer devices added to the pool at no extra cost
• Flexible for more advanced solutions (ex. GPS)
Cons:
• Tester session performance tends to be slower, especially with higher
bandwidth apps like video
• Some security concerns –Testing done outside the firewall, while data
is transferred back and forth through the internet
• Different companies use the same devices with
your app and potentially your private test data
17
XBOSoft, Inc. All Rights Reserved.
18. Using an On-premise Cloud
Types:
• Commercial administration via cart/hub
• DIY lab
Pros:
• Higher performance real-time user sessions
• Increased security over devices and test data
• Allows users to pull devices from lab for hands-on tests
• DIY is least expensive
Cons:
• Limited scalability
• Maintaining devices requires a lab manager
• Procuring/retiring devices can be challenging
• DIY is most difficult to maintain and physically secure
18
XBOSoft, Inc. All Rights Reserved.
19. Emulators
Benefits:
• Most cost effective
• Least infrastructure setup
• Flexible and Scalable
• Support for most test plans
Constraints:
• Cannot execute Usability,
Performance, & Security test
cases on emulators
• Network and hardware
interrupt testing cannot be
done on emulators
Mobile Testing Comparison
Physical Devices in Lab
Benefits:
Covers all test types
Safest option
Should be used for most
common device types
Best for UAT / Usability
Constraints:
Very costly to support across
all OS, device, and version
combinations
Procurement and setup of new
devices can delay testing
Physical Devices in Cloud
Benefits:
Cost effective
Fastest start up
Flexible and Scalable
Can cover approx 90% of test
plan
Constraints:
Latency during usage and
longer test execution time
Lower level of security
excludes this as an option for
some clients
XBOSoft, Inc. All Rights Reserved.
21. Use Real Mobile Devices
the degree to which a test system matches a production
system, including hardware, firmware, devices, software,
appliances, networks, etc.
TEST
INTEGRITY
Emulators/Simulators are tempting due to ease of use and low cost, however:
Proven to cause significant
defect leakage (stats & peers)
Lowest level of test integrity Certain functions cannot be
replicated
21
XBOSoft, Inc. All Rights Reserved.
22. Other Device Lab Considerations
LAB SETUP WILL DEPEND ON:
Test
objectives
on coverage
Manual vs.
automation
Testing types Security Budget Scale
Location of
Teams
22
XBOSoft, Inc. All Rights Reserved.
23. POLL
Approximately how many unique models of mobile
devices exist today?
A. 5,000
B. 10,000
C. 15,000
D. 20,000
E. 35,000
Assumptions:
• We are actually looking for approximate total # of permutations
• Therefore a device with the same model number, but on a different carrier,
counts as more than 1 “model” or instance
• Total active devices in use today globally, including feature phones, tablets,
wearables, but excluding IoT/embedded (cars, kiosks, appliances)
XBOSoft, Inc. All Rights Reserved. 23
24. Test Coverage
Device type (handheld, tablet, kiosk)
Screen size & resolution – dozens just in the top 80%
Device models – 20,000+ unique variations exist!
Operating system – 27 variations in the top 85%
Connection type (USB, WiFi, Bluetooth, Cellular)
Mobile Web vs. App
Variance in carrier installations – 750 carriers globally
Even using 80/20 rule, you are left with thousands of permutations! So what do we do?
24
XBOSoft, Inc. All Rights Reserved.
25. Use Analytics to Understand Your Users
Implicit Analytics: Platform usage and user behavior
Omniture
Google
Analytics
Flurry
TestFlight
(monitor
behavior)
Mixpanel
(native vs.
mobile web)
Localytics
25
XBOSoft, Inc. All Rights Reserved.
26. Use Analytics to Understand Your Users
Xyo
Applause
(external,
sentiment) Appsfire
MeltwaterHootSuite
Explicit Analytics: What your users say
26
XBOSoft, Inc. All Rights Reserved.
28. Customer Feedback: Great place to find those hidden test
scenarios!
1. Social Media (Facebook, LinkedIn Groups, Twitter, etc.)
2. App reviews
3. App ratings / comments
4. Comments, forum posts and articles on the web
5. Complaints and support requests
6. Trending analytics (ex via Applause)
Ensuring Proper Test Coverage – Rounding it Out
XBOSoft, Inc. All Rights Reserved.
29. Lab Definition Exercise
XBOSoft, Inc. All Rights Reserved. 29
First step – review analytics
The data says our customers are:
• 55% on iOS
– 62% iPhone 6, 15% iPhone 6+, 23% iPad
• 43% on Android
– 45% on Samsung S5, 20% on S6, 20% Note 5, 15%
Tab S (tablet)
• 2% on Other (Windows/Blackberry)
30. Lab Definition Exercise
XBOSoft, Inc. All Rights Reserved. 30
Additional considerations:
• 8 screen sizes/resolutions in top 80%
• 12 OS versions in top 80%
• 4 major U.S. carriers
• 1,000 functional test cases
32. Test Coverage Sample By Phase
XBOSoft, Inc. All Rights Reserved. 32
Source: Perfecto Mobile
33. Test Coverage Sample By Index
XBOSoft, Inc. All Rights Reserved. 33
Source: Perfecto Mobile
34. Test Coverage Sample By OS
XBOSoft, Inc. All Rights Reserved. 34
Source: Perfecto Mobile
35. Building a World Class Device Lab
XBOSoft, Inc. All Rights Reserved. 35
36. Overview of Device Lab Components
Desktop Users U.S. or India
Device Management Interface (View device inventory, reserve, etc.)
Real-time Device Viewer (launch sessions)
Remote Control protocol (access to launch apps, perform gestures, text, etc.)
Device interface (USB hub, Desktop/Server, Proprietary Appliance)
Real Devices,
On-premise or Cloud
36
XBOSoft, Inc. All Rights Reserved.
37. U.S. Test
Team
Firewall
Example On-premise Cloud setup
India Test
Team
Mobile Devices
on-premise Cloud
Device access
via desktop tool
Wi-Fi Access Point
Carrier Network
Device access
via desktop tool
Mobile Devices
on-premise Cloud
Wi-Fi Access Point
Carrier Network
Firewall
XBOSoft, Inc. All Rights Reserved. 37
38. U.S. Test
Team
Firewall
Example Public Cloud setup
India Test
Team
Mobile Devices
Off-site Cloud B
Device access
via desktop tool
Wi-Fi Access Point
Carrier Network
Device access
via desktop tool
Mobile Devices
Off-site Cloud A
Wi-Fi Access Point
Carrier Network
Firewall
XBOSoft, Inc. All Rights Reserved. 38
39. Mobile Device Lab Recap
XBOSoft, Inc. All Rights Reserved. 39
• Define mobile for your organization
• Create thorough test objectives
• Weigh all factors when defining lab
requirements
• Decide on Cloud vs. On-premise
• Define test coverage by leveraging analytics
• Monitor channels for continuous feedback
• Use tier-based approach to optimize coverage
40. Post your questions on Twitter and we'll answer them @XBOSoft
Join us to keep updated on all our webinars, reports and whitepapers:
facebook.com/xbosoft
+xbosoft
linkedin.com/company/xbosoft
We post regularly on our blog – check us out! http://xbosoft.com/software-quality-blog/
Download our free Whitepapers, available here:
http://xbosoft.com/knowledge-center/
You will receive an email with information on today’s webinar slides and recording.
Please email us with any further queries regarding our services or ideas for future
webinars! Services@xbosoft.com
Q+A
www.xbosoft.com
XBOSoft, Inc. All Rights Reserved.