In the past few years, Intuit has put an intense focus on ramping up its innovation engine. A big part of this effort is speeding up time-to-market for its hundreds of native mobile apps, while maintaining the highest quality standards. The apps need to work across an ever-increasing number of devices and versions of the operating system.
Once our engineers had access to all possible combinations of devices and operating systems, they needed to be able to kick off automated testing whenever they checked in code. We enable mobile continuous integration by integrating our virtual device lab with our test execution platform and Jenkins system. A notification system detects the arrival of a new build of our mobile products, which kicks off execution of automated tests based on a pre-configured test matrix.
The result is a 95% reduction in the time it takes to test our mobile apps.
Building AI-Driven Apps Using Semantic Kernel.pptx
Velocity Conference: Increasing Speed To Market In Mobile Development Through Continuous Integration
1. Speed to Market in
Mobile Development
Finding the right solution with continuous
integration on real devices
Tina Su, Director of Development, Intuit
2. About me
Nearly 20 years of industry experience in R&D,
quality and engineering management, with a strong
passion for automation.
Engineering Director at Intuit, Developer Productivity
Sr. Engineering Director at Yahoo, Mobile Platforms
Sr. Engineering Manager at Sun, Java Standard and
Mobile Editions
Compiler Developer at MIPS and Tandem
Tina Su, Intuit
3. Agenda
Mobile trends & developers’ challenges
Speed & quality through continuous integration
- Intuit Virtual Device Lab
- Intuit mobile test automation
Takeaways
4. Aggressive mobile momentum
“2013 Internet Trends,” KPCB
Will surpass desktop
by 2014
It should just work!
Proliferation of platforms,
OS versions & device types
5. Developer challenges
Immense configuration matrix
- OS platforms (Android, iOS, Windows, phone vs. tablet, …)
- Inputs (touch/gesture, real buttons, voice, camera, GPS receiver)
- Outputs (portrait/landscape, screen resolution, language)
Application delivery through the markets
- App signing, provision (app & device), submission
Offline experience
- Network, different network bandwidth/latency, no-network
- HTML5 offline app
Media challenges
- Some codecs don’t work on a device
- Streaming, sound
Security & privacy
6. Intuit mobile app portfolio today
Apple
Google
Amazon
SMS
0 5 10 15 20 490
22
15
11
3
491 mobile apps for financial institutions
7. Ramping up Intuit’s innovation engine
Pain: Takes days per build to test
combinations of device/OS configurations
physically & manually, one by one …
QA can’t keep up with the changes
Goal: Reduce development iteration cycle
from days to minutes on an ever-growing
number of devices
Strategy: Speed & quality through mobile
continuous integration
• iPhone 4
• iPhone 4S
• iPhone 5
• iPad 2
• iPad 3
• iPad Mini
Devices
OS versions
• iOS 5.0
• iOS 5.1
• iOS 6.0
• iOS 6.1.x
• iOS 7.0
Builds
X X
Daily
builds
• Apple
• Google
• Amazon
Platforms
X
8. Continuous integration (CI)
What is CI?
Automate build, test & deployment process
Integrate & build early, often, daily, for every check-in
Best practice of Agile development
Jenkins is widely adopted, flexible & extensible
Benefits of CI in a mobile world
Instant feedback on quality, functionality & system impact
Less complex, no integration hell
Detect and address device configuration issues early
Involve stakeholders early
Ultimately achieve shippable release any time
9. Speed and quality
through mobile CI
Key enablers:
• Mobile device lab
• Test automation
Test processorReal
devices
Jenkins
SCM:
source
repository
Developers
/IDE
Automatic
test run
Test
results
Submit the app to
app store if
passed
Check in
Check-in
triggers
automated CI
Install the test app to
real devices in
parallel
Run tests on real
devices in parallel
Analyze
test
results
Trigger testing in
mobile device lab
Process starts here
10. 3rd-party vendor solutions didn’t work for us
Couldn’t handle E2E automation of testing pre-release apps
- Dependency on internal resources & infrastructure
Vendor lock-in: customized CI & test automation solutions
- No flexibility to use our existing tools & frameworks
Disadvantages of vendors’ shared devices
- Security concern on testing on shared devices
- No access to corporate network
- Cost-prohibitive to reserve devices
Private cloud (on-site setup) with local devices not practical
- With sensitive customer data, we need private cloud
- Extensive use of devices for daily testing of hundreds of mobile apps is expensive
11. Why test on real devices?
Most mobile features can be tested on simulators, but simulators
have limitations:
Device capabilities (GPS, camera, accelerometer, gyros), form
factors & pixel density
Real-world experience over the real connection
(3G, 4G, crappy Wi-Fi, etc.)
True performance, security tests
Testing on non-default Web browsers (e.g., Opera Mini)
Not all simulator/emulators have good quality
(don’t reflect real rendering/behavior, bugs)
Testing SMS & app integration
13. Intuit Virtual Device Lab (VDL)
Gives access to real mobile devices using a browser
Intranet
VDL
User machine
User machine
User machine
3-click access
Always on
Integrated with
dev infrastructure
Globally available
iPhone
iPad
Galaxy
Captivate
Optimus Nexus One
Desire
Incredible
Manufacturers/
devices:
Droid
Milestone
Moto
14. VDL architecture – Android
Windows PC
Tomcat
Java image
transferring
application
Browser &
Java applet
Internet/intr
anet
Device
interaction
app
Developer’s laptop
VDL
Image transfer
USB connection
15. VDL architecture – iOS
Mac with Xcode
installed
Tomcat
Java VNC
client
Port
forwarding
Browser &
Java applet
Internet/intr
anet
VNC server
USB connection
Developer’s laptop
VDL
16. Open source tools & technologies used in VDL
Key capabilities Android iOS
Installing mobile applications into
the real device
adb install <APK file> Fruitstrap
https://github.com/ghughes/fruitstra
p
Transferring device screen images,
controlling device
Screencast tool
http://code.google.com/p/androidsc
reencast
Veency VNC app
Fetching device local files adb pull <filename> sftp command
(jailbroken devices only)
17. VDL key features & functionality today
Control the device through standard
browser/RESTful APIs
Change orientation to landscape or portrait
Copy local file to the device from your PC
Get file from the device onto your local PC
Execute an ADB command on the device
See current running logs in your browser
Download history logs to your local PC
18. Continuous integration – how it works
Product
builds
Trigger file launches test
VDL device
inventoryResource
manager
Select a device
Test execution
engine
Results
Monito
r for
new
builds
19. Mobile continuous integration
Leveraging the Virtual Device Lab for test automation on multiple, real devices simultaneously.
Submit, build, test … in minutes vs. hours
Select devices & tests1 Tests run automatically across selected
devices with each build
2 See the results!3
20. Next step: over-the-air replaces USB connection
Fast onboarding
Scalable
Any device
Internet/intr
anet
App
installation
Test
execution
Results
retrieval
VDL
22. Mobile test automation tools
Instrumentation-based
Non-instrumentation
AndroidiOS
MonkeyRunner
MOET
Sikuli
Android SDK
Robotium
TestDroid (BitBar)
Calabash
(LessPainful)
Soasta
iOS SDK
UIAutomation
UISpec
Frank
MonkeyTalk
DA/PerfectoMob
ile
eggPlant
Cross-platform solution
iOS Driver Appium
• Assessed as of 2012
23. Mobile test automation trends
Abstraction with domain-specific language (DSL)
- Common test script over different devices (e.g., Cucumber, MonkeyScript, MOET)
- Support behavior-driven development (BDD)
Client/server-based approach (remotely driven tests via HTTP)
Instrumentation & non-instrumentation solutions complement
each other
Mobile cloud testing environment
- From hardware (cradle) to software approach to drive tests on
real devices
Automated OTA app installation & tests
24. Intuit automation framework of choice/recommendation
Native apps:
- Cucumber + Calabash for BDD on iOS & Android
Mobile Web:
- Selenium/WebDriver
25. What is Cucumber?
It lets stakeholders (customers, QE or developers) describe how
software should behave in plain text.
BDD: Written in business domain-specific language (DSL)
- Good abstraction over different device types
One format for multiple purposes
- UI feature specification
- Automated tests
Cucumber itself is NOT a test automation tool
- Add-on to any test automation framework
- Examples: Calabash, Sikuli & Frank mobile test frameworks
26. What is Calabash?
Open source cross-platform mobile test framework
- iOS + Android, simulator + real device, native + hybrid
Supports Cucumber for BDD
Application needs instrumentation that embeds a Calabash HTTP server
- Tests run from a remote client (in a Jenkins slave)
- Tests can be driven via Wi-Fi (no USB wire)
Ruby clients in API level (Java clients in experiment)
Works great in touch simulation & real device testing
27. Intuit shared test library – rapid test development
Internal open source repository
Teams focus on
creating
Use shared test lib
… and then contribute to
common test library
Tests
Contribute
Tests
Common test library Product tests
Tests
Tests
Globalization test framework
(I18N/L10N)
Performance/load test framework
Security test framework
Native/hybrid test framework
High-level test domain-specific language
(Cucumber steps)
Mobile
back-end
Mobile
front-end
29. VDL cost saving over vendors’ private clouds (today)
Annual
cost ($)
# of devices
Intuit Virtual Device Lab
150
3rd-party vendor 1
local device in cradle
Assumptions: 50 native
apps & daily check-in
Note: As vendor pricing
changes, so do the savings
Savings
25
3rd-party vendor 2
local device w/o cradle
(Software approach)
Savings
Private cloud setup behind corp firewall
30. Key takeaways
Building custom solution vs. 3rd-party vendor solution
Understand your technical/platform/global spread
Many vendor solutions provide a larger set of devices/OS versions
in the global market
Quite a few offer mobile test services – might be tightly coupled with their
device access solution & automation framework
Connecting to your corporate network is a challenge – some provide
private cloud (on-site setup within your network) but it’s cost-prohibitive
Invest in automation and CI
Automate tests with robust automation framework
Enable rapid test development through shared test library
Adopt CI in mobile development for speed & quality
31. More mobile innovations at Intuit
Mobile crash reporter
Mobile components & design patterns
Mobile analytics SDK & gateway
Visit booth #601 – Accelerate Development
Demos & conversations
Contest
Mobile device charging station
Foosball
We’re hiring !
Win a weekend with a TESLA!