6. • Have you ever wrote any kind of UI Tests? Browser,
Mobile, Desktop, etc
• Do you think reliable mobile test automation is possible?
Questionnaire!
8. ➢ Product manager comes up with new features
➢ Developers implement them
➢ QA team test new features and do regression testing
➢ Release
Development flow
9. Whats up?
Development flow ...
Product ManagerQA Team Engineering Team
Where is the
feature?
Code is done.
Waiting for QAToo many features
Too many test cases
Too many devices
No Time
I am dying ...
16. Initial Learning
• Initial learning such as setup, writing simple scenario is not
too difficult.
• With some programming experience one can start writing
mobile e2e tests.
26. ● User Interface issues:
○ UI is not working as per feature specification
○ Test code is not handling UI automation logic properly for scenarios
involving scroll, swipe etc and test is flaky
● Backend issues:
○ Backend is temporary down
○ API is not working as per specification
When can a test fail?
27. When can a test fail? ...
● 3rd party service issues:
○ Service is temporary down and error handling is not being done properly
in test automation logic.
● Hardware issues:
○ Test is failing on specific device because of hardware limitation such as
Memory, Processor.
○ Device wifi is unstable.
28. Problem is not the flaky test!
Problem is we don’t know what is happening
during test execution
34. rspec_html_reporter ...
• Extension of RSpec HTML Formatter with feature such as:
• Pretty report
• Can embed metadata such as screenshot and screenrecord
• ADB Logcat & other metadata are WIP.
35. This report will not solve flaky test itself, but it will help in
distinguishing between a bug and flaky test!
Flaky test solution
36. Let’s go back to difficulties
• Flaky tests
• Test execution environment
38. Test Execution Environment
In general, try to use Cloud Services as much as possible.
This provides quick feedback and they are easily scalable.
And anyone should be able to execute test cases anytime
easily
Android E2E Testing at Mercari
Android Device Farm at Mercari
39. Mercari Android CI (Tools & Service)
• Bitrise: To build application
• Google Cloud Storage: To store built apks
• Google Container Registry: To store docker images
• OpenSTF: For Android Device Farm
• CircleCI 2.0: Build docker image & run tests in parallel containers
• CircleCI Artifacts: To store test reports
• AWS Lambda: To trigger nightly builds
• Slack: Notification & manual triggers
41. • Mobile test automation is challenging but not impossible.
• With correct approach and planning it is possible to create
reliable test automation and actually improve product
quality and release efficiency
Conclusion