This document discusses the challenges of mobile testing and automation. It outlines the complexity of mobile testing due to the large number of devices, operating systems, carriers and resolutions. It also discusses the scope of mobile testing including functionality, usability, load, security and localization. The document recommends automating testing using tools like Perfecto and Jenkins for continuous integration and looping feedback from testing back into the development process. Metrics on app crashes and store ranks are presented and the document concludes with what more can be done, such as increasing test coverage and using tools like Appium.

1. Mobile Testing challenges
Automation
Mobile Automation Meetup , February 2015
Yuval Golan , Erez Nachum , Tali Gilon , Vonage Mobile team

2. www.vonage.com 2

3. www.vonage.com 3
Mobile is “Taking over the world”
It’s a multi platform age with mass adoption.
Where is your Mobile Testing,mobile CI and Mobile
automation?
Complexity Scope Execute Loop back CI Automation

4. www.vonage.com 4
Why is it
so simple
to QA
mobile apps?
Complexity Scope Execute Loop back CI Automation

5. www.vonage.com 5
Complexity
• More than 15K Android devices (last year 4k..)
• Different drivers
• Hundreds of carriers
• Bandwidth & Connectivity
• Tens of OS
• Resolution (LDPI , MDPI…)
• Access rights
• It is actually a phone .. it
vibrate , Peripherals (BT ,earpiece), airplane mode , notifications…
Environment
Device
Code
Complexity Scope Execute Loop back CI Automation

6. www.vonage.com 6
Scope
• Functional
• Usability
• Load
• security
• Localization
• Bugs we wish to resolve
• Network Impairments
• Battery
• Store / purchase
• Disconnections / interrupts (alarm clock ; calls)
Complexity Scope Execute Loop back CI Automation

7. www.vonage.com 7
Execute
• Module
• Level
• Code coverage
• Recent bugs found / fix
• Recent commits / changes
Complexity Scope Execute Loop back CI Automation

8. www.vonage.com 8
How to execute
• Manual
• Automate
• Crowd
• Perfecto / device any..
Complexity Scope Execute Loop back CI Automation

9. www.vonage.com 9
Loopbacks: CI and field feedback
CI Continuous Integration
Pre Commit
QA in most organizations
Field Feedback
Complexity Scope Execute
Loop
back CI Automation

10. www.vonage.com 10
Metrics - Crashes
47% of the
apps crashes
1% of their use
32% of the
apps crashes
2% of their use
Complexity Scope Execute
Loop
back CI Automation

11. www.vonage.com 11
Metrics , Crashes November 2013 – November 2014
Complexity Scope Execute
Loop
back CI Automation

12. www.vonage.com 12
Be alert
listen to the customers and market
• OS (7.1 no audio)
• IPV6 / IPV4 (crash)
Complexity Scope Execute
Loop
back CI Automation

13. www.vonage.com 13
Metrics – Star Ranks / user reviews
Overall Store Rank by Version
Complexity Scope Execute
Loop
back CI Automation

14. www.vonage.com 14
Mobile CI for Commit
Developer commit
some code to the
repository
Jenkins (CI) get
notified and starts
compilation
On success “Static
analysis” and post
commit Code review
running
Deploy an environment
Then automated
“Commit tests”
When commit is
approved it is included
in next QA build , the
latest stable build will
run on nightly
regression
New code version is
distributed to QA/BETA
Complexity Scope Execute Loop back CI Automation

15. www.vonage.com 15
E2E (Seetest) Per commit (KIF +
Espresso)
Automation dev
investment
Write test once for both
platforms
Different code per
platform, duplicate tests
Native actions V (Some IOS issues) X
Scalability License required Open source (dev use it)
Automation project
location
Automation repository Part of Dev repository
Execution time 1m-2m (per E2E test, 2
devices)
0.2m-0.3m (1 device
only)
Reports Built in Requires development
Number of mobile clients
in use
1+ (we use 2) 1
Support V X
Automation 3rd
party Tools that we use
Complexity Scope Execute Loop back CI Automation

16. www.vonage.com 16
Vonage Automation Framework Features• Reporting system (using Jenkins abilities + Junit reports )
• Smart investigation tools (search for specific exceptions and
can connects them to existing Jira tickets, saves investigation time)
• Web Client (Vonage Mobile code compiled on Mac/PC, simulates 2nd
device )
• Automation Test Apps (in order to simulate native changes in native
address book ,native settings…)
• Retry mechanism (differentiates between one timers to
100% reproducible bugs)
• Network traffic investigation tools (Http request- response, SIP flows
in calls , uses app logs)
• Private BE environment (Http request- response, SIP flows in calls, uses app logs)
Our Framework
Complexity Scope Execute Loop back CI Automation

17. www.vonage.com 17
Nightly run
2 tests fail because of an Amazon issue
on uploading large video files.
Good morning
☺
Complexity Scope Execute Loop back CI Automation

18. www.vonage.com 18
Web client
No need in secondary devices for manual/automated tests,
web client behaves like a real client and can even
initiate events to your App.
Complexity Scope Execute Loop back CI Automation

19. www.vonage.com 19
UI – system with Keywords
anyone can
write tests using
it!
Complexity Scope Execute Loop back CI Automation

20. www.vonage.com 20
Module tests
Country Manager module tests we inject input
to the module's functions and compare the output
to the expected results.
Complexity Scope Execute Loop back CI Automation

21. www.vonage.com 21
DDT – Data Driven Test
Send message setup- ~15 lines
Json file with scenarios-
Data Driven
testing
One function +
one “Jason” =
~1K tests
Complexity Scope Execute Loop back CI Automation

22. www.vonage.com 22
What’s next?
•Coverage , coverage , coverage
•Appium
•Appurify
•Qtest
What’s Next ?
Complexity Scope Execute Loop back CI Automation

23. www.vonage.com 23

24. Thank You
Yuval.golan.il@gmail.com