You might not find much information on the topic of large-scale, real-device (phone) test-automation labs, searching the Web. At least, Stephen and his team didn't, and, chartered with designing, building, and maintaining a Firefox OS phone-automation lab of over 40 devices, from the ground up, will share their goals, many challenges, successes, and lots of lessons learned along the way.

1. Building a 40+ real-phone test-automation lab for Firefox
OS:
Goals, Challenges, Accomplishments
San Francisco Selenium Meetup Group
November 19, 2014
Stephen Donner
sdonner@mozilla.com

2. Goals:
● (much) greater capacity
o keep up with b2g-inbound builds for UI + perf (esp. the latter), as much
as possible
● maximal use of resources (devices, limited personnel, etc.)
● multi-node adb mapping/support
● high[er|est] reliability
● easier device troubleshooting/administration
● break new ground and help inform the testing world

3. From this...
(which itself is “test rig 3.0” -- if you can believe that!)

4. 1st Buildout

5. To this...

6. 2nd Buildout

7. Present day
(Bam!)

9. Challenges: (1/4)
● balancing aggressive buildout schedule and challenge with current-rig’s maintenance (a.k.a.
“changing a plane’s engine at 30,000 feet”)
● near-constant task (re)prioritization (a.k.a “new capabilities, new requests/demands”)
● real carriers/networks (AT&T/T-Mobile), real problems:
o spam calls/voicemail
o Amber alerts
o low signal strength/data-connection resets
 slow throughput
 DNS-lookup failures
o logistics of provisioning 40 SIMs
● phone flashing (en masse):
o base (OEM/vendor/reference) build
 non-engineering
● vs.
o daily builds (release candidates, nightlies, etc.)
o aggressive power-management features

10. Challenges: (2/4)
● multi-node adb mapping
o solution had to work with both UI and perf tests
o also required:
 Jenkins-config changes (job/system)
 physical reconfiguration
● and powered hubs
● active SIMs vs. inactive SIMs
o reduce interference/cost/maintenance cost
● DSDS (Dual SIM, Dual Standby)
● SecOps/NetOps requirements
o new office, new problems (new policies)
 basically, new office == slackware of offices ;-)
 port-by-port, host-by-host firewall pass-through
o VPN/Jenkins (CI) access granted on a user-by-user basis

11. Challenges: (3/4)
● RF (Radio Frequency)
o Wi-Fi
 ateam vs. Mozilla Mobile
● capacity + speed + reach
o channels, bands, new router vendor
o FM tuner interference
 requires cabling magic
● ...and perhaps a low-power, local FM transmitter?
o more devices, moar interference?
● physical space/configuration
o shelving space
o power-strip capacity

12. Challenges: (4/4)
● remote teams requested live Air Mozilla streams of the phones, i.e. “what in
the world is the state of that phone in?”
o prototype works, but doesn’t immediately scale bandwidth (# streams,
codecs/encoders)
 informs # of encoder boxes/# of channels
o live (remote) demo
● “keeping the light on”
o camera test failures due to low light
 ...but intermittent

14. Multi-node adb Mapping
Background:
● Jenkins allows many "nodes" to run on 1 host
● ‘node’ meaning ‘agent’
● ADB permits multiple sessions on 1 host
● $ adb -s <serial_num> <command>
Problems:
● Some vendors don't provide a unique serial
● Sometimes OEM (base build) may have same serial number hard-coded in all devices!
o You might not have any control over this :-(
o There are tools for unpacking Android base images and rewriting those serial numbers:
https://github.com/AndroidRoot/BootTools
Solution:
● Create 1 unique node per mobile device
● This = 4 devices to one slave host, 1 per USB port
● Manage scripts - modify our scripts to pass in device serial numbers (to all adb commands)

15. Currently*, we have:
● ~30 active Flame nodes (phones)
o 15 dedicated to UI / 12 perf
o 26 potentially
● attached to 13 active Mac Minis

20. What (was?) is Left?
● MozPool[1] for managing devices, including:
o taking devices offline
o power measurement through in-house, custom-3D-printed ammeters
o remote-reboot capability through power harness
● Puppetization for managing node configurations
[1] https://github.com/mozilla/mozpool

22. Credit Where Credit is Due
● Malini Das
● Raymond Etornam
● Jonathan Griffin
● Andrew Halberstadt
● Dave Hunt
● Johan Lorenzo
● Richard Milewski
● Richard Pappalardo
● Clint Talbert
● Dylan Wong
● Rob Wood
o … and everyone else I forgot!

23. Contact
● Mailing list: mozwebqa@mozilla.org
● Team page: https://quality.mozilla.org/teams/web-qa/
● IRC: #mozwebqa on irc.mozilla.org