Quick overview of our infrastructure for, project genesis, and coverage of Firefox OS (Gaia) front-end UI automation using Marionette (Selenium-backed WebDriver) in Python

1. [B2G | Gaia | Firefox OS] Front-end
Automation in Python (Marionette)
Stephen Donner
Web QA
March 20, 2013

2. Overview
•
Genesis / Introduction
•
Infrastructure (hardware)
•
Software stack
•
Builds in Jenkins / Continuous Integration
•
Test coverage / Capabilities
•
By the numbers
•
Will it blen^H^H run? (demo!)
•
Contribute!
•
Questions?

3. Genesis / Introduction
Initially, “just a B2G QA thing”
http://www.flickr.com/photos/daniel_gies/5041121388/
sizes/z/in/photostream/

4. Genesis / Introduction
Hard to keep good secrets for too long!
...so we expanded coverage + infra
http://memegenerator.net/instance/27635541

5. Infrastructure (hardware)
7 Unagis, tethered via USB to 7 Mac Minis,
running Ubuntu 12.04 LTS
Fun facts:
1) Must have earbuds plugged in for radio tests to
work
2) For the photo/video tests to work, lens/aperture
must not be blocked or in the dark

6. Software Stack
•
Marionette (https://developer.mozilla.org/en-US/docs/Marionette) -
testing server that sits in B2G
•
Selenium-backed WebDriver
•
marionette_client – Python abstraction layer for Marionette
•
Gaiatest (https://pypi.python.org/pypi/gaiatest) - wrapper around
Python unnitests that runs the tests
•
Gaia UI Tests – test framework comprising setUp, tearDown, API
wrappers, and the tests themselves

7. Builds in Jenkins/Continuous Integration
Web QA helps set up, scale, and provide
upkeep for more than just our UI tests
http://qa-selenium.mv.mozilla.com:8080/view/B2G/

8. By the Numbers
●
Started on November 11, 2012
●
25 individual contributors [1]
●
From Taipei, Romania, UK, Canada, USA, Beijing
●
89 [2] tests, consisting of:
●
~ 19 unit tests
●
80 UI tests
●
~1hour per build, including initial flash
[1] https://github.com/mozilla/gaia-ui-tests/graphs
[2] http://qa-selenium.mv.mozilla.com:8080/view/B2G/job/b2g.unagi.gaia.nightly.ui/lastBuild/

9. Test coverage / Capabilities
Sampling of coverage:
•
Dialer (make/receive calls, test IMEI #)
•
Contacts (call, delete, create, add photo to, send SMS to)
•
Send/receive SMS
•
Camera (add, delete photo/videos)
•
Clock (add/delete alarms)
•
Calendar (add event, assert today's date)
•
Browser (load pages over EDGE, Wi-Fi, bookmark, navigate back)
•
Play a YouTube video
•
Marketplace (search, install, load, delete apps)
•
Everything.me (browse to and load/install an app)
•
Changing wallpaper
•
Radio (adding/remove station, dial frequency up/down)
•
Keyboard
•
Notification bar
•
and more...

10. Test coverage / Capabilities
Capabilities:
•
we push to device (media files – photos, contacts, etc.)
•
we also capture the app HTML, take screenshots on failure for post-test debugging, and
store the latest run's logcat (full, debug)
•
we test the default/empty states
•
in-between tests, we use the devicemanager API to iterate through and delete media
files, delete all contacts, clear Wi-Fi networks, restore modified settings to default, set
volume to 0
•
manifest.ini
•
gives us the ability to tell which tests to run, where
•
attributes:
antenna, b2g, wifi, lan, camera, xfail, panda, sdcard, qemu, bluetooth
•
•

11. Challenges
•
Two moving targets: B2G (Gaia/Gonk) + Marionette
•
Marionette-enabled builds are not identical to non-engineering builds
•
different online/offline handling (patch pending)
•
changes to how/when adb / remote debugging work in Gaia
•
memory leak(s)
•
feature work
•
performance refactoring
•
Event-model changes:
•
.click() → .tap()
•
async JavaScript – team's expertise is (still) largely Python
•
very dynamic, heavy HTML5 apps, with CSS3 usage – rotates, transforms, SVG – hard to
find and test location and state of the app
•
infrastructure set up
•
acquiring Unagi phones (and keeping them healthy)
•
SIMs (unlimited plans)
•
figuring out and maintaining a distributed, consistent format to store each phone's info

12. Will it blen^H^H^ (demo!)
https://www.youtube.com/watch?v=CC2ywYeP-4s

13. Contribute!
•
Repository:
•
https://github.com/mozilla/gaia-ui-tests
•
How to contribute:
•
https://github.com/mozilla/gaia-ui-tests/blob/master/CONTRIBUTING.md
•
Open issues up for grabs!
•
https://github.com/mozilla/gaia-ui-tests/issues?
direction=desc&labels=difficulty+beginner&sort=created&state=open
IRC:
•
•
#mozwebqa (core Gaia UI Tests)
•
#ateam (perf tests)

14. Questions?