1. 1 16. Mai 2014
Customer Visual
Best Practices in Mobile App Testing
Daniel Knott | @dnlkntt
Vienna Mobile Quality Crew
Auf die Plätze, Fertig, APP!

2. Who am I ?
• Daniel Knott
• Software Test Manager @AOE
• Mobile Testing since the end of 2010
• Before AOE, Team Lead QA @XING Mobile Team
• Blogger and Speaker
http://www.adventuresinqa.com
@dnlkntt
http://www.linkedin.com/pub/daniel-knott/1a/925/993
https://www.xing.com/profile/Daniel_Knott

3. 3 16. Mai 2014
I Company Profile

4. Intro
Current Status
Mobile Challenges
The Customer
Mobile Specific
Testing
Security
Tools

5. 5 16. Mai 2014
I Current Status
Author: Barry Caruth, https://www.flickr.com/photos/caruth/8401943928/

6. Current Status | Some Numbers
2+ millions apps available
100+ billion downloads
80% – 90% of apps are deleted after the first usage
56% of apps that requires a registration gets deleted
Average user checks device 150 times per day
45% are not well tested

7. 7 16. Mai 2014
I Mobile Challenges

8. Mobile Challenges | All known!?
http://opensignal.com/reports/fragmentation-2013/

9. Mobile Challenges | All known!?
http://opensignal.com/reports/fragmentation-2013/

10. Mobile Challenges | How to handle fragmentation
Group your devices, A B C
Test only on those devices
Test on real device
Don’t use simulators or emulators
Know the target group

11. Mobile Challenges | How to handle fragmentation
Group 1, Prio A
• High End
Devices
• Dual/ Quad Core
CPU
• RAM >1024MB
• Retina, Full HD
Display > 4,5“
Group 2, Prio B
• Mid-range
devices
• Dual Core CPU
• RAM <1024MB
• Screen size < 4“
• Software not
older than 1 year
Group 3, Prio C
• Small devices
• Single Core
• RAM < 512MB
• Low screen size
and resolution
• Older OS, older
browser
Based on your target group

12. Author: Davide Vizzini Limbte, http://www.flickr.com/photos/limbte/4007115435/
Mobile Challenges | More…
Author: Juan de Dios Santander Vela, http://www.flickr.com/photos/juandesant/209098053
Author: Rain Rabbit, http://www.flickr.com/photos/37996583811@N01/8033259053/

13. 13 16. Mai 2014
I The Customer
Author: Michael Newman, https://www.flickr.com/photos/mzn37/1226016289/

14. The Customer | Better know them
Know the target group
Talk to your customer
Will downsize testing effort
Do usability testing with them
Invite them to a beta testing program (Sonos, FB,
Twitter)
Author: Grzegorz Lobinski, http://www.flickr.com/photos/gregloby/3515990945/

15. 15 16. Mai 2014
I Mobile Specific Testing
Author: Theus Falcao, http://www.flickr.com/photos/theusfalcao/9563767145/

16. Mobile Specific Testing
Known functional testing is still required
• Boundary values
• Equivalence classes
• Testing against acceptance criteria
Test the app as black box
• Try to break it
• Be creative and think in edge cases

17. Mobile Specific Testing
Author: Juan de Dios Santander Vela, http://www.flickr.com/photos/juandesant/209098053
Author: Ian Britton, http://www.flickr.com/photos/freefoto/728651045/
Author: Moyan Brenn, https://www.flickr.com/photos/aigle_dore/5626341059/
Author: @Doug88888, http://www.flickr.com/photos/doug88888/7149187135/
Author: Dan Levy, http://www.flickr.com/photos/danlevy/5607259711/

18. Mobile Specific Testing
Author: amateur_photo_bore, http://www.flickr.com/photos/andresv/233667770/ Author: Rain Rabbit, http://www.flickr.com/photos/37996583811@N01/8033259053/
Author, Daniel Sancho, http://www.flickr.com/photos/teclasorg/2852716491/

19. Mobile Specific Testing
Author: Surat Lozowick, http://www.flickr.com/photos/suratlozowick/4311105154/ Author: Surat Lozowick, http://www.flickr.com/photos/suratlozowick/4464587302/
Author: Theus Falcao, http://www.flickr.com/photos/theusfalcao/9563767145/

20. Mobile Specific Testing
Android
• JouleUnit, to measure battery usage
• Monkey, to stress test the app
iOS
• Energy Usage in Instruments
• UI AutoMonkey, to stress test the app

21. Mobile Specific Testing
Author: Molly Holzschlag, http://www.flickr.com/photos/mollyeh11/272434559/ Author: Quinn Dombrowski, http://www.flickr.com/photos/quinnanya/4718635484/

22. Mobile Specific Testing
Author: minxlj, http://www.flickr.com/photos/minxlj/3270996158/ Author: Ryan B. BulaPhotography, http://www.flickr.com/photos/nayralub/3763581778/
Author: Sterling Coffey, http://www.flickr.com/photos/n0ssc/7046915929/

23. 23 16. Mai 2014
I Tools
Author: OZinOH, http://www.flickr.com/photos/75905404@N00/7126147125/

24. Tools
Do test automation!
Start simple and discover different tools
Don’t buy huge test automation tools
No tool is perfect
See which tool fits in your development environment

25. Tools | Android
Robotium (ExtSolo)
Selendroid
Spoon
Calabash for Android
Robolectric
Espresso
Monkey
Great mobile tools provided by the company square: http://square.github.io/
MonkeyTalk
Appium

26. Tools | iOS
UIAutomation
MonkeyTalk
Keep It Functional
Calabash for iOS
ios-driver
Kiwi
XCTest
Appium

27. 27 16. Mai 2014
I Security
Author: Alexandre Dulaunoy, http://www.flickr.com/photos/adulau/7712545428/

28. Security
Author: Anthony Catalano, http://www.flickr.com/photos/badwsky/34164244/
Author: Dennis Wong, http://www.flickr.com/photos/denniswong/2406135310/
Âuthor: orkomedix, http://www.flickr.com/photos/orkomedix/5211437958/

29. Security
Security testing is not easy
Get help from experts
Check the OWASP mobile security page
https://www.owasp.org/index.php/OWASP_Mobile_
Security_Project
Think twice about app permissions

30. 30 16. Mai 2014
I Summary

31. Mobile Specific Testing | Summary
Combine your in house testing with crowd testing
Be up to date! Know your target group
Keep the mobile features in mind
Do test automation and know the right tools
Do security testing
Be creative and on the move while testing mobile apps

32. Questions?
Thank you for your attention!
http://www.adventuresinqa.com

33. 33 16. Mai 2014
AOE GmbH
LuisenForum, Kirchgasse 6
65185 Wiesbaden
Germany
Phone: +49 6122 70 70 7 - 0
Fax: +49 6122 70 70 7 - 199
E-Mail: sales@aoe.com
Web: www.aoe.com
I in Germany

34. Links & References
• World Quality Report [http://www.sogeti.com/en-us/looking-for-solutions/Services/Software-
Control-Testing/The-World-Quality-Report-2013--2014/]
• Robotium [https://code.google.com/p/robotium/]
• Spoon [https://github.com/square/spoon]
• Robolectric [http://robolectric.org/]
• Calabash for Android [https://github.com/calabash/calabash-android]
• Selendroid [http://selendroid.io/]
• ExtSolo
[http://docs.testdroid.com/_static/extSolodocs/com/bitbar/recorder/extensions/ExtSolo.html]
• Espresso [https://code.google.com/p/android-test-kit/wiki/Espresso]
• Monkey [http://developer.android.com/tools/help/monkey.html]
• UI AutoMonkey [https://github.com/jonathanpenn/ui-auto-monkey]

35. Links & References
• MonkeyTalk [https://www.cloudmonkeymobile.com/monkeytalk]
• UIAutomation[https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/
InstrumentsUserGuide/UsingtheAutomationInstrument/UsingtheAutomationInstrument.html]
• Keep It Functional [https://github.com/kif-framework/KIF]
• XCTest
[https://developer.apple.com/library/ios/documentation/ToolsLanguages/Conceptual/Xcode_Overvi
ew/UnitTestYourApp/UnitTestYourApp.html]
• Calabash for iOS [https://github.com/calabash/calabash-ios]
• ios-driver [http://ios-driver.github.io/ios-driver/?page=home]
• Kiwi [https://github.com/allending/Kiwi]
• Appium [http://appium.io/]
• JouleJunit [https://code.google.com/p/jouleunit/]
• Energy Usage Instruments
[https://developer.apple.com/library/ios/documentation/AnalysisTools/Reference/Instruments_User
_Reference/EnergyUsageInstrument/EnergyUsageInstrument.html]