As organizations implement their mobile strategy, testing teams must support new technologies—while still maintaining existing systems. Melissa Tondi describes the major trends and innovations in mobile technology, usage patterns, tools, and test equipment that you should consider when transitioning existing test teams or starting new ones. Based on information from more than two years of research with a lab-based consultant team, Melissa focuses on areas that balance efficiency and productivity, including use of a Device Matrix technique to select devices to test against, and when to use emulators and simulators rather than physical devices. She offers solutions to ensure you have a comprehensive mobile test strategy and focuses on challenges—including understanding mobile-specific integration testing and which automation tools to use—that have inundated traditional test teams. Melissa describes how to build a well-organized device lab, incorporating testing scenarios—such as gesture and interruption testing—that are unique to mobile.

1. T14
Track Session
4/21/2016 2:00 PM
"Innovations in Mobile Testing:
Expanding Your Test Plan"
Presented by:
Melissa Tondi
Denver Automation and Quality Engineering
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 · 904-278-0524 · info@techwell.com · www.techwell.com

2. Melissa Tondi
Denver Automation and Quality Engineering
Melissa Tondi has spent most of her career working within testing teams-
concentrating on functional, performance, security, and mobile testing
techniques. In her current roles, she is the founder of Denver Mobile and Quality
(DMAQ) and, as Head of SQE at ShopatHome.com, she is assisting teams to
continuously improve the design, build, test, and delivery of quality software.
Throughout her career in the software test and quality engineering fields she has
focused on organizing testing teams around three major tenets—efficiency,
innovation, and culture. Melissa’s previous roles have included Vice President of
Professional Services for a Mobile-focused startup, Director of Software Quality
Engineering in the world’s leading education company; QA consultant for health
care, finance, and software-as-a-service industries; and president of the Software
Quality Association of Denver.

3. 4/8/16
1
Mobile Dev+Test 2016 - Innovations in Mobile
Testing: Expanding Your Test Plan
Melissa Tondi
Takeaways and Agenda
→ Level Setting: Your current (or new) mobile testing strategy
– Understanding your Mobile Solutions
• Mobile Web: RWD and AWD
• Apps: Native and Hybrid, Near Native
• IoT (a brief definition)
→ Expanding your Test Plan - Test Types that Mobile has
Introduced
→ Expanding your Test Plan - Accessibility
→ Expanding your Test Plan - Using Simulators and Emulators
→ Device proliferation and using the Device Matrix
→ Environments and Tools – Proxies, Performance, CI

4. 4/8/16
2
Level Setting - Understanding your Mobile
Solution: Mobile Web
What is “Mobile”? Mobile web sites designed to match every web-
enabled device. They’re accessed through the mobile device’s web
browser and do not need to be downloaded/installed. They re-direct to
an M Site or similar.
– Responsive Web Design – an approach aimed at building sites to
provide an appropriate viewing experience (reading and navigation
with minimal resizing, panning, and scrolling) across a wide range of
devices. It adapts the layout to the viewing environment.
• No matter how much you resize the screen, that same layout will
automatically respond to that size
• A single codebase for a website can serve both mobile and desktop
users. So instead of having two separate versions of your site (one
desktop site and one mobile site), you will have a single responsive
site that will accommodate simultaneously mobile and desktop
users
Level Setting - Understanding your Mobile
Solution: Mobile Web
What is “Mobile”?
– Adaptive Web Design – uses the server to detect the device that’s
being used. A separate template is maintained for each device that
is supported. There are several distinct layouts for multiple screen
sizes and the layout used depends on the screen size used.
• The goal is a highly differentiated experience that is built for the
specific intent of the mobile customer
• It allows you to specifically capture user intent more precisely on a
mobile device

5. 4/8/16
3
Level Setting - Understanding your Mobile
Solution: Mobile Web
Responsive Web Design Pros and Cons
– Harder to build initially, but is flexible as new devices and IoT
(Internet of Things) expand users’ experiences
– Images are scaled down rather than resized or eliminated
altogether therefore, they generally take longer to load
Adaptive Web Design Pros and Cons
– Faster to initially build, but requires maintenance as new devices
are supported
– Because they are created for mobile users they have smaller and
optimized images. Images are loaded to meet the device resolution
and screen size, making page load time more efficient.
Level Setting - Understanding your Mobile
Solution: IoT
A Brief Definition…
IoT (Internet of Things): a network of physical objects that feature an
IP address for internet connectivity and the communication that
occurs between these objects and other Internet-enabled devices and
systems
→ Our current, common IP Addresses are based on IPv4. It uses 32-bit
addresses and is limited to around 4.3 billion addresses
→ IPv6 is slated to replace IPv4 and uses 128-bit addresses with 3.4×1038
unique addresses of capability

6. 4/8/16
4
Level Setting - Understanding your Mobile
Solution: Things to Know
RWD and AWD
→ Use CSS (Cascading Style Sheets) to adjust layout
→ The spots where it changes are called “breakpoints”
→ Functionality may be hidden, or workflows may be adjusted based upon
size
→ Usually adjusted to vertically scroll
Tools
→ BrowserStack
– https://www.browserstack.com/
→ Mobile-Friendly Test
– https://www.google.com/webmasters/tools/mobile-friendly/
→ responsive test
– http://responsivetest.net/#u=http://www.zootemplate.com|1024|768|1
→ Responsinator
– https://www.responsinator.com/
Level Setting - Understanding your Mobile
Solution: Things to Know
IoT
→ Gray Box Testing
– Understanding the Architecture, OS, Third Party Hardware, Apps,
Firmware, Connectivity Protocols
→ Real Time Operating System (RTOS)
– Operating Systems that serve real-time application process data as it
comes in, typically without buffering delays
→ Service Virtualization
– Emulates only the behavior of the specific dependent component.
Instead of virtualizing an entire database, you monitor how the
application interacts with the database and then replicate the related
database behavior.

7. 4/8/16
5
Level Setting - Understanding your Mobile
Solution: Apps
Native Apps: Are coded with a specific programming language. They
are fast, reliable, and powerful, and are tied to a mobile platform and
downloaded from a central portal by OS (i.e. Apple App Store, Google
Play). That means you must duplicate them using the appropriate
programming language in order to target another mobile platform.
These apps are written in the language specified by the platform
→ Objective-C or Swift for iOS using the iOS SDK (Software
Developer Kit)
→ Java for Android using the Android SDK
Level Setting - Understanding your Mobile
Solution: Apps
Hybrid Apps: These rely on development frameworks and are
written with the same technology used for websites and mobile
web implementations. They are hosted or executed inside a
native container on a mobile device. They ensure cross-
platform compatibility and can access the phone’s hardware
(camera, GPS, user’s contacts, etc.)
These are regarded as web applications wrapped inside of a
mobile app. The mobile app wrapper itself uses a web viewer
(a scaled down version of a browser) to display the web pages.
Web pages are developed with the same technologies that are
used for both traditional and mobile web like:
– CSS
– HTML
– JavaScript

8. 4/8/16
6
Level Setting - Understanding your Mobile
Solution: Apps
Near Native: These have business logic core written in a language
other than what is provided by the platform. The UI and other aspects
make full use of native components and APIs.
The result is an app that can be targeted to multiple platforms but run
natively on each.
There is a somewhat high percentage of code reuse across platforms.
Example: Apps written in Appcelerator's Titanium. They are generally
written in Javascript, but make use of all native components.
The app can be targeted to multiple platforms but it runs natively on
each
Level Setting - Understanding your Mobile
Solution: Things to Know
→ Apps require much greater hardware testing
– 3D graphics
– Device sensors
→ Apps need to work on all screen sizes
– Avoid wasted space
→ Apps define XML layouts similar to HTML
– Node Structure
– Static Content (Images)
– Dynamic Content (Color, text)

9. 4/8/16
7
Level Setting - Understanding your Mobile
Solution: Operating Systems
Android:
→ Custom version of Linux
→ Apps built in java
→ Chrome Browser
(WebKit)
→ Over 20,000 different
Android devices
→ Custom Manufacturer
code
→ Forked versions of
Android (like the Kindle)
iOS:
→ Proprietary Apple OS
built on BSD (Berkeley
Software Distribution)
→ Only works on Apple
hardware
→ Apps built in Objective C
and Swift
→ Safari browser (WebKit)
→ Less than 20 different
devices
→ Low OS variance
Expanding your Test Plan – Test Types
Peripheral
– Wireless Testing – NFC (Near Field Communication), Bluetooth/
Bluetooth LE Accessory, Stylus
– Wired
– Internal to Device – Headphone Jack, Keyboard
– External to Phone - CC Readers, Bar Code Scanners
Connection Testing
– USB Power/Data
– 4G/LTE/CDMA+/Wimax
– 3G/GSM/CDMA, 2G/Analog, WiFi, Hotspot Generation
– Computer Tethering
– Carrier
– Throttle – Limiting bandwidth and measuring an App’s performance

10. 4/8/16
8
Expanding your Test Plan – Test Types
Gestures
– Swipe, tap, double tap, drag, pinch, spread expand, flick, shake,
orient, tilt, press, press and hold, press and tap, press and drag,
swipe and hold, rotate.
Interruption Tests
– Controlled - Plug in/out USB, power, and headphone, Home Button,
Power Button, Navigating to another App, Save State
– Uncontrolled/”Elevator” test - SMS, Phone Call, Notifications
Expanding your Test Plan – Test Types
Syncing
– Updating two or more locations to ensure applicable files are
equivalent.
– App to Cloud, App to Computer, App to App, Device to Device
Internal Hardware Integration
– Camera, GPS, Accelerometer, Battery Drain, SIM Card, Volume,
SMS, Microphone, Speakerphone

11. 4/8/16
9
Expanding your Test Plan – Accessibility
Worldwide Web Consortium (W3C) ​
– http://www.w3.org/TR/WCAG20/#guidelines​ - Web
https://www.w3.org/TR/wcag2ict/ - Mobile
You’ll want to determine which Level is appropriate for your
organization and users, but here’s the gist:​
– Level A – A good place to start. Great for organizations that already
have a product in use and who want to establish a baseline for
accessibility conformance.​
– Level AA – The next step. This level means that most people will be
able to use your site/product in most situations. Many education
and government agencies require this level.​
– Level AAA – The most difficult to achieve and maintain. In rare
situations, this may be required, but the W3C makes it clear that it
is not possible to satisfy all Level AAA success criteria for some
content.​
Expanding your Test Plan – Accessibility
Exercise – Android
Settings > My Device >Accessibility > Vision >
TalkBack > Turn TalkBack on
Exercise - iOS
For iOS phones, go to Settings > General >Accessibility >
VoiceOver > Turn VoiceOver on

12. 4/8/16
10
Expanding your Test Plan – Accessibility:
Things to Know (Android)
Android Gestures
All gestures for TalkBack require only 1 finger.
→ Touch or move around the screen – speaks and select the item under
your finger
→ Swipe right or down – move to next item on screen
→ Swipe left or up – move to previous item on screen
→ Double-tap – select focused item
→ Swipe up then down – move to first item on screen
→ Swipe down then up – move to last item on screen
→ Swipe right then left – scroll forward on page or move slider
→ Swipe left then right – scroll back on page or move slider
→ Swipe up then left – Home button
→ Swipe down then left – Back button
Expanding your Test Plan – Accessibility:
Things to Know (iOS)
iOS Gestures
→ 1 finger gestures:
– Touch or move around the screen – speaks and select the item under
your finger
– Tap 2 times fast – activate the selected item
– Swipe left or right – selects the previous or next item
– Swipe up or down – move to the selected rotor option
→ 2 finger gestures:
– Swipe up – read all from first object in selected area
– Swipe down – read all from selected item
– Scrub back and forth – go back, cancel, close pop-up
– Pinch out/in – select/deselect
– Rotate to open the web rotor and select rotor items (see the last point
for 1 finger gestures)
→ 3 finger gestures:
– Swipe left or right – move to previous or next horizontal screen/page
– Swipe up or down – move/scroll down or up

13. 4/8/16
11
Expanding your Test Plan– Emulators,
Simulators
Choosing Emulators and Simulators
– A lab is not complete without a mixture of both
• A Simulator mimics only the software. iOS has one, but it
does not simulate any of the iPhone hardware layer
• An Emulator mimics both software and hardware. Android
has one, but only emulates CPU and Memory
Expanding your Test Plan– Emulators,
Simulators
When to use Simulators/Emulators
→ Testing the Function
• Broken buttons, missing images, correct formatting for vertical
and horizontal modes
• RWD/AWD or mobile site elements display correctly
→ Previewing the Form
• Demo basic design and layout mockups before significant
development investment is spent on polishing the UI
• See the application within the frame of a real device, compared
to the window of a development program

14. 4/8/16
12
Expanding your Test Plan – Emulators,
Simulators
When to use Simulators/Emulators
→ Accelerating Development Time
• For simple visual checks, opening an app in a device emulator
can be as simple as switching windows, compared to fully
loading a physical device
→ Test more frequency with quick checks on an emulator, thus finding
bugs faster and with greater context
Expanding your Test Plan – Emulators,
Simulators: Things to Know
Simulators
16-Bit Display
Operating System
User Interface
Emulators
Keyboard and Buttons
WiFi
Simulated SIM Card
System on a Chip
Sound Chip
Flash Memory

15. 4/8/16
13
Expanding your Test Plan – Device Testing
Things to Know
24-Bit Display
Touchscreen
Multi-Touch
Camera & Video
Voice Recognition
True Sound Quality
USB Connections
Bluetooth
SD Cards
Network Data
Real Phone Calls
Dual SIM Cards
Battery State
Device Temperature
Real GPS & Sensors
Pre-Installed Apps
App Store Access
App Installation
Email Sync
Contacts Sync
Calendar Sync
App Log Encryption
Network Security
Fingerprint Scanner
Build Your Lab

16. 4/8/16
14
Build Your Lab – Getting Started
Determining your Device Matrix
→ Operating System
– OS customizations, missing libraries, driver issues
→ Screen Size
– Rendering issues, usability, missing layouts
→ Pixel Density
– Density Independence, missing layouts
→ Aspect Ratio
– X,Y calculations, overlapping panels, display issues
→ System on a Chip (SoC)
– Hardware performance, Instruction set, battery, signal
→ Carrier
– Network protocol, speed, responsiveness, packet loss
Build Your Lab – Getting Started
Determining your Device Matrix – The Goal
→ Efficiency, not necessarily 100% coverage
→ Build your lab to cover both apps and mobile web
→ Define a list of categories of devices
→ Adjust the number of devices based on reasonable coverage

17. 4/8/16
15
Build Your Lab – Getting Started
Determining your Device Matrix – Example Categories
→ Common: Matches the Most Common Display Configuration
→ Newest: Latest OS, Largest Screen, Highest Resolution
→ Oldest: Oldest, smallest, slowest device
→ Abnormal: Non-standard OS, Aspect Ratio, Orientation, and
Size
→ Popular: In Terms of Sales
→ Budget: Low-priced New Model
→ Flagship: By Manufacturer (for example)
– iPhone 7 (TBD)
– Galaxy Note
– Google Nexus
– LG G5
→ Catch-All
Building your Lab – Android
Category Device Name OS Size Density Resolu8on DPI AR Protocol System on Chip
Newest Sam. Galaxy S 4.2.2 Normal xhdpi 1920x1080 441 16:9 LTE/GSM Qualcomm S4
Flagship LG Nexus 4 4.2 Normal xhdpi 768x1280 318 3:5 GSM Qualcomm S4
Oldest HTC TaNoo 1.6 Small ldpi 320x240 143 4:3 GSM Qualcomm S1
Popular Sam. Galaxy S3 4.1.2 Normal xhdpi 1280x720 326 16:9 CDMA Samsung Exynos
Common Mot. Droid 3 2.3.4 Normal hdpi 960x540 275 16:9 GSM Cortex A9
Abnormal LG OpTmus VU 4.0 Large xhdpi 1024x468 256 4:3 GSM Nvidia Tegra 3
Budget Dell Venue 2.2 Normal mdpi 480x800 228 3:5 GSM Snapdragon S2
Catch-all Sony Xperia P 2.3 Normal hdpi 960x540 275 16:9 GSM Sony NovaThor

18. 4/8/16
16
Building your Lab - iOS
Device
Name
OS Display Aspect SoC Carrier
Newest iPhone 6 8 5.5” 1920 x 1080 ppi 9:16 A8 Chip with *
MoTon
Coprocessor
T-Mobile
Oldest iPhone 3g 6 3.5” 320 x 480 165ppi 2:3 Apple A3 AT&T
Common iPhone 5s 7/8 4” 1136 x 640 326ppi 9:16 Apple A5 Verizon
Popular iPhone 6+ 6 3.5” 640x960 330ppi 2:3 Apple A4 Sprint
iPad
(Re8na)
iPad Air 7 9.7” 2048 x 1536 264ppi 3:4 Apple A7 Verizon
iPod iPod Touch
(4th gen)
5 3.5” 640x960 326ppi 2:3 Apple A4 WiFi
Mini iPad Mini 8 7” 1024 x 768 162ppi 3:4 Apple A5 AT&T
Build Your Lab – Summary
Determining your Device Matrix – End Result
→ Logically Select Devices
→ A Large Number of Permutations can be Covered with a
Handful of Devices
→ As New Devices are Announced, Run them through the DM
to Determine whether to Purchase

19. 4/8/16
17
Environments
Test Environments
– Browserstack.com or SauceLabs.com
• Online Simulators for any OS or mobile device
• Useful for troubleshooting and regression testing
• Defects aren’t always reproducible on real devices
• Automation Cloud (supports WebDriver)
• Quick rendering comparisons across multiple devices
– Hardware Rack
• High Power USB Hub necessary to keep tablets charged
• Stands to maintain device orientation
Tools – Things to Know
Mobile Automation Tools - Noteworthy
→ Mobile automation is still in its infancy
→ Mobile devices lack good automation hooks
→ Most tools can only control an app
→ Many automation tools require modification of the app
– Custom libraries, recompilation, or external
application
→ Many automation tools can’t automate the stock web
browsers
– safari, chrome
→ Most tools can’t automate outside the app (OS,
configuration, etc.)

20. 4/8/16
18
Tools - Open Source vs “Closed”
→ Open Source is
customizable, and
maintainable by anyone
trained
→ Open source tools
typically require a
custom framework
→ Anyone with a coding
background can work on
the tests
→ WebDriver is the W3C
standard for web
application testing
→ Closed tools are more
usable for those without
development
backgrounds
→ Closed tools are
expensive
→ Closed tools offer better
reporting and
integrations out of the
box
→ Closed tools mean it
may be hard to find
people with experience
in specific tools
Tools - Open Source
Open Source Options
→ Selenium
– Multiple side projects support mobile testing
– Selenium experience is very common
→ Appium, Android Driver, Robotium
– Appium being merged as part of Selenium 3.0
– Acts as a remote webdriver node
– Run tests on simulators or physical devices (USB)
– Don’t have to change the app
– Can automate default browsers (chrome / safari)
– Can automate android / iOS apps
– Same libraries used for desktop automation
– Used in Browserstack and SauceLabs
→ MonkeyTalk
– GUI based
– Selenium based
– Sort-of Free and Open source

21. 4/8/16
19
Tools – “Closed”
→ EggPlant
– Image based
automation tool
– Can control full device
(not just apps)
→ HP UFT
– Large Market Share
– GUI Based
– Expensive
→ Ranorex
– GUI Based
– Supports android 2.2+
→ TestComplete
– GUI or Code driven
– Multiple language
choices
– Also supports windows
applications
Other Tools
Proxies
→ A proxy intercepts all HTTP traffic from a computer or
mobile device
→ Traffic can be inspected, paused, re-executed, and
manipulated
→ Valuable for debugging purposes
→ Can be included with defect reports
→ Similar to chrome dev tools
→ Examples
– Fiddler (windows)
– Charles (Mac)
– BrowserMobProxy (Selenium)

22. 4/8/16
20
Tools – Other Considerations
Performance Testing
No such thing as client side performance testing
→ You can do memory, battery, CPU, RAM profiling during
normal testing activities
→ Generating Server load
– HTTP Traffic
→ Most load testing tools will support mobile
– Configure it as a proxy
– Record traffic
– Parameterize
– Generate Load
Tools – Other Considerations
Continuous Integration
→ A build server (Jenkins, TeamCity, Bamboo, CircleCI) kicks
off tests periodically
→ Can view pass/fail status, troubleshoot, etc.
→ Can be hooked up with most automation tools
→ Tests can be executed against physical devices, local
simulators, or cloud simulators
→ Speed up time through parallel execution
→ Setup a local GRID cluster for faster results
→ High power USB hub for tablets

23. 4/8/16
21
Contact Information
→ email: melissa.tondi@gmail.com
→ Twitter: @melissatondi
→ Blog: MelissaTondi.blogspot.com
→ LinkedIn: Melissa Tondi