While Unit and Integration testing in mobile software is a great way to help maintain quality and produce stable software, testing user interaction until more recently has been done solely by humans. System testing for Mobile isn't a new concept and yet is very hard to accomplish. System Testing with Robotium can help Android Developers shorten the feedback loop between QA and will help developers to start thinking the way users will actually use their software. My presentation discusses the why, what, when, where and how of System Testing with Robotium.

1. © mutualmobile
SYSTEMTESTING WITH ROBOTIUM
Why,What,When,Where and How

2. © mutualmobile
WHO AM I?
Elliott Chenger
Android Engineer @ Mutual Mobile
@echenger erchenger
• Likes
• Long walks on the
beach
• Everything Android
• Ice Hockey
• Auburn
• Cars
• Dislikes
• the F word
(fragmentation)
• when carriers delay
software updates
• King Joffery from
GOT
• I-35 traffic

3. © mutualmobile
AGENDA
• Brief overview of different types
ofTesting
• What value does system testing
add?
• Different tools for system testing
• What we chose and why.
• Do’s and Don’ts to system
testing
• How to write system tests
• Live Demo
• Q&A

4. © mutualmobile
DIFFERENTTYPES OFTESTING
White Box
• Tests code for precision and
correctness
• Tester understands and is testing
internal structure and design of
software
• Generally Unit and Integration testing
Black Box
• Software internal structure is not
known to the tester
• Mainly done through acceptance and
system testing
• Tests are based on requirements

5. © mutualmobile
WHAT IS SYSTEMTESTING?

6. © mutualmobile
SYSTEMTESTING
• Subset of Black BoxTesting
• Tests software as a whole to make
sure it meets specified requirements
• Defined by acceptance criteria
• Mimics the human

7. © mutualmobile
WHY SHOULD A DEVELOPER WRITE
SYSTEMTESTS?

8. © mutualmobile

9. © mutualmobile
SYSTEMTESTINGTOOLS
• Robotium
• Espresso
• MonkeyTalk
• UIAutomator

10. © mutualmobile
WHY WE CHOSE ROBOTIUM?
• Written in Java and structured like JUnit
tests
• Bootstrapping was easy
• Allows us to write tests that mimic
what a human does
• Strong community support

11. © mutualmobile
PROCESS/ACTIVITIES
• The project kicks off and a full audit of all current system testing tools is performed.
• After a month of reviewing various tools Robotium becomes the clear selection among all of the
stakeholders.
• A machine was purchased to be used as our testing server.
• The script and boiler plate for testing was created.
• Through trial and error our script and the server became more stable.
• We switched from emulators to physical devices and we decided that Flow testing will be more
beneficial instead of screen testing.

12. © mutualmobile
• Worked with QA to identify how we are going to capture tests and
communicate what should and shouldn’t be tested.
• We started a pilot program for Robotium on two major projects.
• Much was learned about how we actually write these tests.
• Now we are going through the process of educating and advocating our
department to add system testing into our normal development process

13. © mutualmobile
LESSONS LEARNED

14. © mutualmobile
AUTOMATE
Let Continuous Integration systems do
the heavy lifting for you.This will help you
catch issues before it gets into QA’s
hands.

15. © mutualmobile
USE PHYSICAL
DEVICES
We have found through testing that
Android emulators are simply slow and
unreliable.Another benefit is being able
to test on different device subsets.

16. © mutualmobile
WRITETESTSTHAT
MIMIC A HUMAN
Put yourself in the user’s shoes, don’t get
hung up testing pieces of the UI directly.
Thinking this way makes your tests more
stable and more valuable to QA and your
customer.

17. © mutualmobile
DON’T STRIVE FOR
100% COVERAGE
Strive for good tests, not a certain
quantity of test coverage. Beware of
writing tests just to have tests. Like most
things SystemTesting has a time and a
place.

18. © mutualmobile
SCREENTESTING
VS
FLOWTESTING

19. © mutualmobile
SCREENTESTING

20. © mutualmobile
FLOWTESTING

21. © mutualmobile
CODE
The fun part

22. © mutualmobile
OUR EXAMPLE APP

23. © mutualmobile
AGENDA
• Setting up Robotium in a project
• Organizing your tests
• Boilerplate code
• Writing tests
• RunningTests
• Live Demo

24. © mutualmobile
SETTING UP ROBOTIUM WITH ECLIPSE
• Download jar
https://code.google.com/p/robotium/

25. © mutualmobile
SETTING UP ROBOTIUM GENERIC
• Download jar
https://code.google.com/p/robotium/
• Add the jar to your classpath

26. © mutualmobile
SETTING UPTHE MANIFEST

30. © mutualmobile
BOILER PLATE

38. © mutualmobile
WRITINGYOU FIRSTTEST

39. © mutualmobile
BEFORE WE START DO’S AND DON’TS
• DO write tests that mimic a human.
• DO write your tests to encompass the overall acceptance criteria
• Assert DO NOT assume (also follow assertions with detailed error
messages)
• DO NOT depend on time.
Example: Don’t assume a REST call will always take 2 seconds.

40. © mutualmobile
LET’STEST A STORY
• User Story:As a user I would like to be able to login to the app, so that I can see the next
screen
• Acceptance Criteria
• Button for logging in
• A place for the user to enter their name for logging in.
• Once they enter their name and click the log in button they go to the next screen.
• If they don’t enter a name they get an error dialog

53. © mutualmobile
SO FAR
• We have written tests for the happy
path for testing login.
• We have an Activity that has visuals but
nothing else in it.

54. © mutualmobile
LET’S NOW RUNTHETESTS

58. © mutualmobile
NOW CODEYOUR FEATURETO MAKE
THETESTS GO GREEN

61. © mutualmobile
NOW WE CAN WRITE SOME SAD PATH
TESTS

65. © mutualmobile
NOW
• We have written happy path tests and
they are successful.
• We just finished writing our sad path
tests and they should not pass.
• We have an Activity that has visuals and
some of our interactions but we still
need to code the rest of the sad path
test.

66. © mutualmobile
NOW CODEYOURTESTSTO GO GREEN
ANDYOU ARE DONE!

67. © mutualmobile
LIVE DEMO

68. © mutualmobile
WHATS NEXT?
• Go explore: https://code.google.com/p/robotium/
• Download the example app code for this presentation:
http://goo.gl/WPkwNE
• Have questions or feedback?
elliott.chenger@mutualmobile.com or @echenger

69. © mutualmobile
Q&A