Mobile phones are available since the middle of the 1980s. Since then, the devices changed savagely but the biggest change happened in 2007, when the first iPhone was presented by Apple. Since then, the mobile smartphone market knows only one direction – UP! Since 8 years touch devices are everywhere, from smartphones to tablets.
More than 2 millions apps are available for download in the stores of the biggest vendors and this number is still increasing. There are apps for photos, music, games, office and many more categories just to name some of them.
But what about the quality of those apps? Are those apps reliable, trust worthy, easy to use, well developed and tested? The latest world quality report from Sogeti shows that almost half (45 percent) of mobile apps are not well tested in terms of functionality, performance and security.
This workshop includes insights into the challenging job of mobile testing from native to web apps. Best practices will be provided to become a better mobile tester. Besides that, this workshop will show different test techniques from functional to non functional mobile testing, test automation tools and how to handle the device fragmentation.
The workshop will not include practical mobile test automation. However, the participants will be able to test different mobile apps manually based on the content of the workshop.
2. Who am I ?
• Daniel Knott
• Software Test Manager/ Mobile Tester @AOE
• Software Tester since 2008
• Mobile Testing since the end of 2010
• Author, Blogger and Speaker
http://www.adventuresinqa.com
@dnlkntt
http://www.linkedin.com/pub/daniel-knott/1a/925/993
https://www.xing.com/profile/Daniel_Knott
5. 5 26.März2015
I Current Status
Author: Barry Caruth, https://www.flickr.com/photos/caruth/8401943928/
6. Current Status | Some Numbers
2+ millions apps available
100+ billion downloads
+19,000 Android Devices
16 iOS Devices
7. Current Status | Some Numbers
80% – 90% of apps are deleted after the first usage
56% of apps that requires a registration gets deleted
Average user checks device 150 times per day
45% are not well tested
8. Current Status | Mobile Testing is different
High User expectation
Lot’s of different devices with HW and SW
Users are on the move
Mobile Tools and Approaches
12. Challenges| How to handle fragmentation
Group your devices, A B C
Test only on those devices
Test on real device
Don’t use simulators or emulators
Know the devices of your target group
13. Challenges | How to handle fragmentation
Group 1, Prio A
• High End
Devices
• Dual/ Quad Core
CPU
• RAM >2048MB
• Retina, Full HD
Display >= 5“
Group 2, Prio B
• Mid-range
devices
• Dual Core CPU
• RAM <=1024MB
• Screen size <= 4“
• Software not
older than 1 year
Group 3, Prio C
• Small devices
• Single Core
• RAM < 512MB
• Low screen size
and resolution
• Older OS, older
browser
Based on your target group
14. Challenges | The User - KIFSU
Simple UseableFastItKeep
K I F S U
15. 16 26.März2015
I How to break your app
Author: OZinOH, http://www.flickr.com/photos/75905404@N00/7126147125/
16. Mobile Specific Testing
Known functional testing is still required
• Boundary values
• Equivalence classes
• Testing against acceptance criteria
Test the app as black box
• Try to break it
• Be creative and think in edge cases
17. Mobile Specific Testing
Author: Juan de Dios Santander Vela, http://www.flickr.com/photos/juandesant/209098053
Author: Ian Britton, http://www.flickr.com/photos/freefoto/728651045/
Author: Moyan Brenn, https://www.flickr.com/photos/aigle_dore/5626341059/
Author: @Doug88888, http://www.flickr.com/photos/doug88888/7149187135/
Author: Dan Levy, http://www.flickr.com/photos/danlevy/5607259711/
18. Author: Davide Vizzini Limbte, http://www.flickr.com/photos/limbte/4007115435/
Mobile Specific Testing
• Logout (English)
• Ausloggen (German)
• Déconnexion (French)
• Çıkış yap (Turkish)
• Выйти (Russian)
19. Mobile Specific Testing
Author: amateur_photo_bore, http://www.flickr.com/photos/andresv/233667770/ Author: Rain Rabbit, http://www.flickr.com/photos/37996583811@N01/8033259053/
Author, Daniel Sancho, http://www.flickr.com/photos/teclasorg/2852716491/
20. Mobile Specific Testing
Author: Surat Lozowick, http://www.flickr.com/photos/suratlozowick/4311105154/ Author: Surat Lozowick, http://www.flickr.com/photos/suratlozowick/4464587302/
Author: Theus Falcao, http://www.flickr.com/photos/theusfalcao/9563767145/
21. Mobile Specific Testing
Author: Molly Holzschlag, http://www.flickr.com/photos/mollyeh11/272434559/ Author: Quinn Dombrowski, http://www.flickr.com/photos/quinnanya/4718635484/
22. Mobile Specific Testing
Author: minxlj, http://www.flickr.com/photos/minxlj/3270996158/ Author: Ryan B. BulaPhotography, http://www.flickr.com/photos/nayralub/3763581778/
Author: Sterling Coffey, http://www.flickr.com/photos/n0ssc/7046915929/
23. Author: Anthony Catalano, http://www.flickr.com/photos/badwsky/34164244/
Author: Dennis Wong, http://www.flickr.com/photos/denniswong/2406135310/
Âuthor: orkomedix, http://www.flickr.com/photos/orkomedix/5211437958/
Mobile Specific Testing
25. Exercise II
Build up groups 2-5 person
Download a mobile app of your choice
Test it with provided scenarios
Note down the bugs
30 Minutes
Let’s talk about the bugs
26. Exercise II – My Bugs – Android Wordpress App
Using the app in bad network
conditions
Don’t show technical errors
27. Exercise II – My Bugs – Android Wordpress App
Provide user friendly error messages
Test app without backend
availability
28. Exercise II – My Bugs – Android Wordpress App
Language size
Test your app in different languages
29. Exercise II – My Bugs – Android Wordpress App
Localization problems
30. Exercise II – My Bugs – Android Wordpress App
Localization problems
32. Mobile Specific Testing | Summary
Mobile Testing is different compared to other software
Be creative and on the move while testing mobile apps
Be up to date! Know your target group
Keep the mobile features in mind
Keep the high user expectations in mind
33. Thanks for your attention!
www.handsonmobileapptesting.com
More information about mobile testing
can be found in my book
„Hands-On Mobile App Testing“
Soon available in a printed copy, too.
Welcome to my workshop
My Name is Daniel Knott
Who is new to mobile testing?
This workshop contains some talking time from my side roughly around 30minutes and the rest of the time we will have some exercises.
Please raise your hand whenever you have a question
Test Automation will not part of this workshop, there is not enough time for that. However if you need some mobile testing tools just ask
Currently software test manager @AOE, Mobile Testing for a big german telecommunication provider. Testing iOS, Android and BlackBerry Apps
Mobile Tester since end of 2010
Build up the mobile testing team @XING
Mobile test automation for iOS and Android
Get in touch with me over the provided social links
IT agency since 1999
150 employees, HQ Wiesbaden, SFO, Zürich
Background TYPO3 and Magento
Own Search product ‚Searchperience‘
Enterprise e Commerce shops e.g. Angry Birds Webshop, QVC
Enterprise CMS Systems: Congstar
Sony Entertainment System
ConnectedCar, Mobile Apps
Enough marketing! Lets get started
Intro shows the topic I will cover today in this presentation
For each topic I will give some best practices
Lets get startet. If you have questions just ask them
I have slides for 30minutes. Remember if you have any kind of question don‘t hesitate to ask them.
16 iOS devices (7 iPhones von 4 – 6+ | 9 iPads von iPad2)
Some number about the mobile world, numbers based on different reports e.g. Gartner, Compuware and World Quality Report
Every 6 minutes an AVG. User is checking his smartphone, thats more than 150 times a day!
- Users also expect fast loading times under 2 seconds!
Keep this numbers in mind
Number from Gartner, Comupware and World Quality Report
Why is mobile testing different?
High user expectations, reminder the 90%
Mobile Testing is different
Remind the numbers from prev. Page
- This was a rough first overview of the mobile world.
10 Minutes
I will also my current challenges with possible solution for it, maybe you have the same challenges.
20 min
Device Fragmentation
User expectations -> Remember the huge numbers from the first slides.
Data Networks
Mobile Usage
Device Specific Functions
Tools
10min
In the projects I was working on we used the following approach to downsize the effort in device fragmentation
Lets have a look at the grouped devices
Group Devices into A B C, example for techi app. Customers always have the latest devices.
Group or grading is different for every app
Create groups based on your target group
Keep in mind, this is an ongoing process!!!
Handling the devices in an mobile team could be a full time job!
Don‘t underestimate the efforts here.
If you don‘t want to buy phones, you can rent them. Open Device Labs, Cloud Devices!
Based on KISS Principle -> Keep It Simple Stupid
I created my own mobile principle KIFSU to keep user expectation in mind
Follow the 3 simple rules Fast, Simple and Useable and your customer will be happy
Best Practices
Know your target group.
With that knowledge you can downsize the dev and testing effort. Because you know the devices of your customers and their needs
Talk to them! Invite them to your company and talk to them (not all of course :D)
Then do usability testing with them
Invite them to be a beta tester. For example I tested the Sonos App in a beta program. Was a nice approach to feel valuable as a customer
Create Personas based on your target group
20min
I think you all know the biggest mobile challenge! You can hear this in nearly every mobile report, post, article
Software and Hardware Fragmentation, different CPU, RAM, Camera, Battery, Screensize, Density
iOS Fragmentation is also there!
Two new form factors with the iPhone6 and iPhone 6+
But also keep the different Software Versions in mind
But there are more challenges than just the fragmentation
But how to handle the fragmentation?
The is also a challenge when developing a mobile app!
If you don‘t know your customer you don‘t know what to develop and on which platform to test
Why is the customer here a big challeneg? Any ideas? Because Mobile users have a much higher expectation to mobile apps than to any other software like browser based applications
Get to know the customer
- Lets come to mobile specific testing.
First of all, the known testing techniques are also valid for mobile apps
Like boundary testing, equivalence classes, acceptance criteria
In the mobile challenge slides I already mentioned the fragmentation, language and the network
Be mobile and on the move while testing your app.
Test your app in the real environment!
Test in data networks!
To handle network scenarios do Mobile Field Testing
Test the app in the city, in the country side, while commuting to work, in the park, at home be mobile
Test the app also in the environment the app is made for. When testing sports app, do sports :)
When testing e.g. a skiing app, be on the slope!
Network situations are hard to automate. Simulating them is ok but not enough
Test also more network provider
If you don‘t have the time for doing that kind of field testing, use the crowd! They can offer you that kind of stuff.
Plane Mode
Edge, 3G, 4G
Keep different languages in mind
Logout (English)
Ausloggen (German)
Déconnexion (French)
Çıkış yap (Turkish)
Выйти (Russian)
Test the device specific hardware
1. Camera. Different Vendors == different cameras and resolutions. Even in iPhones different cameras are installed
2. Test the sensors that are used by your app. Tilt sensors, motion sensor, light sensor, shake the phone etc.
GPS testing, check that used GPS is released by the app again to save battery
- TWITTER APP EXAMPLE
3. Memory, test if your app can be stored on sdk card on android. Test how your app behave when the local phone storage empty, full is no crashes? Proper error message?
Put lots of data to your device and see how it behaves
1. Test the battery consumption of your app.
Charge the app before testing to 100% and mesaure every 10 minutes the current battery state
Use tools for measure the battery. There are some tools on the market.
Also use an nearyl empty phone for testing and see how the app is performing. Some phones switch off some sensors to save battery, can your app work with that?
2. Start the app and leave the phone in standby mode, open the app again, is the app still working? Maybe triggering a reload to get the latest data from the backend
Start the app from multitask switch off the network plane mode!?
3.Test different interrupt notifications, incomming sms, mms, phone call usw. If your app supports push notifications open the app and send a push notification check the handling. _> monkey tools
Use tools like JouleUnit (Android)
On iOS use the Energy Usage Function of Instruments
Monkey for Android to stresstest your app and see the handling of the interrupts
UI AutoMonkey for ios
There are some tools that help you analyzing the battery consumption of your app. And tools to stress test your app
Use tools like JouleUnit (Android)
On iOS use the Energy Usage Function of Instruments
Monkey for Android to stresstest your app and see the handling of the interrupts
UI AutoMonkey for ios
On this slide there is one category that most of the people don‘t care. But in some apps it makes really sense.
1. Do usability testing
Invite people to test your app
Starbucks test
Test early versions, prototypes, skribbles
2. Think of handycaped people (blind) and do some accessibility testing (screen readers, color testing)
Use the build in screenreaders! At least try to think at those people
In my current project the iOS app needs to fully accessable. This was really hard to develop and to test. You need to think of the contrast and the colors of the app. Every part had to be tested with a screenreader and so .
Keep this point in mind or try it tomorrow
1. Test the update process of your app
Check that user is still logged in
Check that data is still the same
Check that database changes will not affect the current data
Test the update from an older version to the current version
2. Do performance testing
Hard to do manually, but compare at least the old with the new version in terms of loading time (gut feeling is always good)
If there is time and money see how to use an performance testing tool to measure the app (enterprise software)
3. Check the log files of the app while testing
Often there are exceptions
Check that the request and response is encrypted
Lock for debug information that should not be in the release candidate of your app
1. Check if the communication between your app and the backend is encrypted (use a proxy tool)
Check that the data that is stored is encrypted in the local database
2. Check that the signing certificate is valid, check signing password of the app
Check also the developer account password from the stores!
3. Use only the permissions you really need in the app
Ask the devs twice what they have done!
30min
20min
Summary, lets just recap the last slides
10min
We have seen pleenty of topics that needs to be covered when a mobile app should be tested
Here are some other facts and point that maybe help you guys in your daily work
Thank you very much for your attention.
If you have questions feel free to ask them
If you are still looking for mobile testing content visit my blog from time to time, I try to keep you updated there!
Thats it so far for the mobile specific testing
Lets come to mobile tools.
When I talk to other mobile testers, this is always a problem. The tools and to find the right mobile test automation tool.
The tools based in the next couple of slide are open source tools
Before I start with some tools, I want briefly talk about the test automation pyramid
You see the traditional test automation pyramid for software applications
But this pyramid is not applicable for mobile apps
Mobile Testing requires lots of manual testing
Unit Testing for mobile apps is not easy and sufficient
Lots of interfaces or APIs needs to be mocked or is not possible
Integration testing is not easy
UI Tests including E2E testing is important
Mobile tools are not mature enough for unit testing
Most projects look like this from a test pyramid point of view. It‘s flipped
No stable foundation, it can fall to the right or left side
Therefore I created a mobile test pyramid
Due to the current tool situation and the fact that mobile requires lots of manual testing has flipped the testing pyramid
Foundation is manual testing
Followed by End to end automation from the UI to the backend systems
New is Beta Testing. Have the customer expectations in mind. Use beta testing in your project. Either with colleagues or using a crowd testing approach
And still use unit tests but this time very little compared to the rest.
NEW: The pyramid contains manual and automated parts
I use this pyramid in my mobile projects.
Before I come to some tools I wrote down some sentences that maybe helps you finding a mobile test automation tool.
1. First of all, DO TEST AUTOMATION.
It is not easy, but you have to do it, also on mobile
2. Don‘t huge test automation tools, enterprise tools might be good, but most the time there are just expensive and slow. For example, when there is a new update from ios or android it could be that the software is not supporting it
Since 4 years I am working with open source tools and this worked out pretty good!
Open source can be extended, by YOU!
3. Start small and simple. KISS is the magic word here. Discover the tools, see the documentation and so on
4. No tool is perfect neither closed or open source.
Mobile test tools are still at the beginning
5. See which tool fits best in your dev and testing environment! Thats different for every company.
But now, some tools for Android and iOS
List some open source test frameworks for android Robotium Recorder
List is not complete
Most of the tools simulate real user interaction
All frameworks except Robolectric to execute the tests on the real device!
You can combine tools
A good way to find the right tool is to build a dummy app that contains all elements your future app will have, then test the test tool and see if the tool can handle all the elements
A short example from my current project. Android project, want to try espresso (test tool from google). We have a date and timepicker in our app. Espresso has no support for that elements
so we are using robotium together with spoon
A greate source for mobile tools is the dev corner from the company square. And there are many other tools on the market.
Also a list of open source tools
Android offers more open source tools than on the iOS side
All listed tools are automating the user interface
Most of them can execute the test on the real device
Sometime it make sense to combine the tools, like Kiwi or XCTest with Keep It Functional
There are also tools in the cloud that could be used for testautomation, there are plenty of providers
Waiting for Selenium 3, this should offer full mobile test automation features
Another tip that was always pretty helpful in my mobile projects, if the mobile test tool is using the same language than the production app. Then developers and testers can work together on the test environment.
- Lets come to the last topic for today
- Lets have a look at security testing on mobile devices.
Security testing is not easy
Get help by experts to avoid image loss, money and whatever
To get started with the topic security testing check the OWASP Mobile security page