Mobile testing presents a daunting challenge to software development shops and testers. QA/test teams must design test plans to account for multiple platforms, an enormous and ever-increasing number of devices, and frequent OS updates that often introduce dramatic changes. To add complexity, the increasingly consumer-centric demands of the field require new features and fixes—almost daily. Although test automation is often cited as the best—if not the only—possible solution for mobile, it has presented some big challenges. Tasked with compressing the testing process for FIS Mobile, the leading mobile financial transaction platform, Steven Winter and his team revamped their approach to quality by implementing precision mobile test automation, continuous integration, and on-site mobile labs. This discussion will overview how the team reduced potentially months of manual mobile testing to an overnight automation task. The objective of the discussion is to take back new ideas to identify the core requirements of continuous mobile testing and overcome hurdles to mobile quality.

1.
W5
Session
4/15/2015
11:00
AM
"Continuous Mobile Testing for
Critical Business Apps"
Presented by:
Steven Winter
FIS Mobile
Brought
to
you
by:
340
Corporate
Way,
Suite
300,
Orange
Park,
FL
32073
888-­‐268-­‐8770
·∙
904-­‐278-­‐0524
·∙
sqeinfo@sqe.com
·∙
www.sqe.com

2. Steven Winter
FIS Mobile
Steven Winter loves building teams that break big things! As the director of
quality for FIS Mobile, Steven is responsible for the complete quality ecosystem
for America’s leading provider of mobile financial services and the innovative
power behind the Starbucks Card Mobile. With nineteen years of experience in
quality, Steven has tested through a wide array of technologies from web,
firmware, hardware, cameras, phones, desktop, data—and everything in
between. He has continually pushed the testing edge and delivered high
performance, world-class QA operations at Broderbund, First Data, Leapfrog,
PureDigital/Flip, mFoundry, and now FIS Mobile.

3. 4/8/15
1
CONTINUOUS MOBILE TESTING FOR
CRITICAL BUSINESS APPS
Steven Winter
Director of Quality, FIS Mobile
Speaker Bio
Director of Quality for FIS Mobile responsible for the complete quality ecosystem for
America’s leading provider of mobile financial services, including over 1,200 banks with
over 2,000 apps in market, and the innovative power behind the nations first and most
successful mobile payments initiative, the Starbucks Card Mobile. With 19 years of
experience in quality, Steven has successfully delivered high performance, world-class
QA operations to a wide array of companies like Broderbund, First Data, Leapfrog,
and PureDigital/Flip, mFoundry, and now FIS Mobile.
Steven Winter
Director of Quality
FIS Mobile

4. 4/8/15
2
• World’s largest banking and
payments provider
• Nation’s largest mobile banking
and payments provider
• > 3000 Mobile Apps in market
• +1,600 banks, +29MM end users
• Most successful payments app
• Most successful banking app
• Award-winning mobile wallet
FIS Mobile
FIS Mobile earned the 2013 Best
in Class Mobile Banking Vendor
Overall from Javelin Strategy &
Research, the group’s highest award
for mobile banking vendors.
AGENDA: KEY TAKEAWAYS
• Challenges of being an early leader in mobile quality
• Core requirements of a new approach
• Mobile test automation capabilities
• Role of continuous integration in mobile delivery
• Real benefits in speed and quality

5. 4/8/15
3
MASS
MOBILE
ADOPTION
INCREASING
DEVICE
DIVERSITY
RAPID
CLIENT
GROWTH
CONSUMER
DRIVEN
INNOVATION
INCREASING
SECURITY
EXPLOITS
INCREASED
TESTING
COMPLEXITY
NEED
MORE
DEVICES
UNABLE
TO FINISH
TESTING IN
1 SPRINT
THOUSANDS
OF APPS TO
SUPPORT IN
MARKET
SCALABILITY
AND SPEED
PRESSURE
CAUSES
EFFECTS
What were our big challenges?
What are challenges of being an early leader
in mobile quality?
OPERATIONAL TECHNICAL
• Immaturity of mobile test
automation
• Deciding what to automate
• Regression “Tsunami”
• MDM – Mobile Device
Management
• Staff
• ROI expectations
• Diverging delivery teams
• Mobile testing is difficult
• Device compatibility matrix
• Lack of robust test tools
• Reliable test results
• Configurable script
framework
• Technical debt

6. 4/8/15
4
Reducing the Regression Tsunami
Efficient Scripting Architecture
Single, configurable scripts base used by all teams
C ore
Product
Integrat ion an d
Depl oy ment
Hoste d
Ser vices
UNIT
API
DEVICE

7. 4/8/15
5
What are the core requirements of a
new approach?
ü Vision, Goals, Metrics
ü Robust Script Architecture
ü “Divide and Conquer” Approach
ü Real Devices
ü Need the Right Tools for the Job
ü Immediate Results to the Developers
ü Tight Control of Test Noise
How do we test?
Test Framework Tools/Platform Primary Usage
Unit • jUnit
• Fast, code-level test writing and execution
• Granular testing from unit test level upwards
Server API
• Java
• Apache HTTP Client
• Web services testing of server API
Client UI
• EggPlant
• TouchTest
• Calabash
• Selenium (SWAP)
• Requirements-based testing of UI flow
• Testing against real devices and emulators
Performance
• CloudTest
• Jmeter/BlazeMeter/New Relic
• In-depth performance runs
• Targeted small-scale tests executed frequently
Security
• Fortify
• IBM AppScan
• NowSecure
• ViaLabs (in-house tool)
• Static code analysis
• Dynamic server analysis
• External scans done by vendors
• Internal scans done by security tester

8. 4/8/15
6
FIS Mobile CI System
SVN JENKINS
Build
Deploy
Test
Results
Report
STEP 1
STEP 2
STEP 3
STEP 4
STEP 5
What is the current state of mobile test
automation capabilities?
Object Recognition
On-device Performance Telemetry (RUM)
Enhanced Mobile Performance Testing
Maturing Test Platforms and Support of CI
Device Clouds
1
3
5
2
4

9. 4/8/15
7
What is the role of continuous integration
in mobile?
Cease dependence on mass
inspection to achieve quality.
Improve the process and build
quality into the product in the
first place.
- W. Edwards Deming
“
”
What is the role of continuous integration
in mobile?
• Essential to compete in time-to-market and scalability
• Represents a huge paradigm shift
• Provides faster, more specific test results so developers
can work smarter
• Identifies integration issues sooner
• Enables continual quality

10. 4/8/15
8
What are the real benefits in speed and quality?
• 250 hours of manual tests = just over 1 hour every night
• Nightly Regression Tests instead of waiting until feature complete –
4 months to overnight
• 25% reduction in time to market for production deployments
• Earlier bug fixes
• Declining field failures (aka “Escapes”)
• Lots of ancillary tools as a result of developing test and CI frameworks.
One tool alone saved 40+ hours a week!
SUMMARY
• Mobile test technology is catching up
• Very real benefits from mobile CI
• We are freeing the humans!

11. 4/8/15
9
Q & A
THANK YOU
Steven Winter
steven@fismobile.com
@winterweb