Lessons learned about QA can manage the rising Android device fragmentation

1. www.immobilienscout24.de
Android Fragmentation
QA lessons learned from the fragmentation hell
Android Meetup Berlin | 29.08.2012 | Jürgen Schmidt

2. About me
Jürgen Schmidt
www.immobilienscout24.de
Test Engineer
Development Lifecycle Expert
Database addicted

3. Fragmentation
QA must consider different
levels of fragmentation
• OEMs (>27)
• OS levels (16)
• Display size/resolution (>100)
• Further hardware features
 Devices (> 400)
Page 3 | Android Fragmentation | Jürgen Schmidt

4. Strategy
Follow the famous 80/20 rule
20%
• With an effort of 20%
effort
cover 80% of all devices to test
• Source of usage statistics:
• App statistics on publisher 80%
•
site on Google Play Store
Reporting Frameworks like
coverage
Google Analytics
• Don‘t forget potential new devices or features
Page 4 | Android Fragmentation | Jürgen Schmidt

5. ImmoScout App Facts… OEM
Samsung SonyEricsson HTC
Which devices are currently LG Motorola Google
used by our users? Sonstige
Top30 ~80%
20%
1%
Display Resolution
2%
Find a minimum set 3% 49%
480x800 720x1280 320x480 480x854
among top devices 540x960
8%
720x1184 800x1232 800x1280
which cover all top 240x320 Other
17%
80% device features 2%~ 92%2%
2%
2%
3%
4%
8%
9% 56%
12%
Page 5 | Android Fragmentation | Jürgen Schmidt

6. Selected top devices for manual test
Wildfire S Galaxy S 2 Galaxy S 3 Nexus 7
OEM HTC Samsung Samsung Asus Google
OS level 2.2 (8) 2.3.5 (10) 4.0.3 (15) 4.1 (16)
Display Res. 320x480 480x800 720x1280 800x1280
Density ldpi mdpi hdpi mdpi
Page 6 | Android Fragmentation | Jürgen Schmidt

7. Test automation
We consider OS level fragmentation for running unit tests
and robotium system tests on virtual devices
Jenkins (CI server) with Android plugin offers multi-
configuration jobs for executing tests on different virtual
devices at the same time
Page 7 | Android Fragmentation | Jürgen Schmidt

8. Further tests to help finding bugs
caused by fragmentation
Session Based Tests
• Unstructured exploratory test
driven by test and domain experts
• Use complete device pool
Crowd-sourced Testing
• Bug hunting by real users
under real conditions
• Many testers with a lot of
various devices
Page 8 | Android Fragmentation | Jürgen Schmidt

9. Monitoring a (pre-)published app
Many real-time bug reporting frameworks can be integrated
into the app
Help in evaluation of reported exceptions or crashes by
grouping and annotating of device features
Page 9 | Android Fragmentation | Jürgen Schmidt

10. Conclusion
Quality risks caused by fragmentation are managable
Experience show: Most found critical bugs are not specific
for certain devices, OS level etc.
Focus only on top and new features instead reaching full
coverage
Limit your efforts in testing fragmentation!
Page 10 | Android Fragmentation | Jürgen Schmidt

11. Want to read more about fragmentation?
Android fragmentation visualized by OpenSignalMaps:
http://opensignalmaps.com/reports/fragmentation.php
TechCrunch article to show how several game developers
manage fragmentation
http://techcrunch.com/2012/06/02/android-qa-testing-
quality-assurance/
About Session Based Testing by Satisfice Inc.:
http://www.satisfice.com/sbtm/
Blog post about Crowd-sourced testing by Mithun Sridharan
http://it.toolbox.com/blogs/mithuns-memoirs/crowd-
sourced-software-testing-52763
Page 11 | Android Fragmentation | Jürgen Schmidt

12. www.immobilienscout24.de
Thanks for
your attention!
Contact:
ImmobilienScout24 Fon: 030/24301-11 00
Andreasstraße 10 Email: info@immobilienscout24.de
10243 Berlin URL: www.immobilienscout24.de