At Badoo, we run more than 1200 end-to-end tests for our main iOS application. This should amount to more than 40 hours of tests, but we run them in just 1.5 hours. And now, thanks to some updates, we've got it down to a mere 35 minutes. In this talk, I will describe how we moved from tests tightly coupled to infrastructure (simulator hosts and tools) to an iOS remote device server. I'll also explain how this simplified our parallel test runs and made our tests and infrastructure easier to maintain and scale.

1. Remote iOS Device Server
Scaling iOS testing
Nikolai Abalov

2. Hello, my name is
Nikolai Abalov
Senior Automation Engineer at Badoo
Previously Senior Developer and Automation Engineer
at 2GIS for 3 years
Co-creator of Winium.Mobile
From Siberia
github.com/NickAb
@NickAb

3. Agenda
1. Introduction
2. Parallel tests on iOS
3. Device Server
4. Results & next steps
5. Know-Hows — how to apply it at your scale
6. Recap
3

4. Introduction
4

5. What we had
● 10 Pull Requests per day to the app repo
● 1200+ end-to-end tests
● 35+ hours of cpu-time
● 1.5 hours wall-clock time
● 1500+ cpu-time hours per day
● Extra projects on the same infrastructure
● Tightly coupled tests and infrastructure
5

6. What we want
● People trusting the automation
● Fast and reliable feedback
6

7. What we want
● 30 minutes wall-clock
● Stability
● Scalability
● Easy maintenance and development
7

8. How to get there
● Delete ⅔ of tests
8

9. Questions?

10. Joking aside
● Filter tests based on change set, coverage, components
● Speed up and stabilise tests
● Heavily parallelise test runs
● Provide scalable infrastructure to run tests
10

11. Parallel tests on iOS
11

12. Test parallelisation
1200 tests
∕ 1 simulator
= 40 hours
1200 tests
∕ X simulators
= 0.5 hours
X = 80 simulators
12

13. Options
● 3rd
party cloud
● SauceLabs / TestObject
● Xamarin Test Cloud
● AWS Device Farm
● In-house
13

14. Multiple users, simulator per user
Agent
Rake / Parallel Cucumber
ssh qa1@h1 ssh qa2@h1 ssh qa1@h2 ssh qa2@h2
macOS,
Xcode,
Ruby,
…
14
Test Test Test Test
iOS Calabash in parallel on one host by Tim Baverstock

15. Single user, multiple simulators
Agent
Rake / Parallel Cucumber
ssh qa@h1 ssh qa@h1 ssh qa@h2 ssh qa@h2
15
Test Test Test Test Test Test
iOS Parallel Automation: run faster than fast by Viktar Karanevich
Turning iOS End to End Testing up to 11 by Lawrence Lomax
macOS,
Xcode,
Ruby,
…

16. Issues with current model
● Tight coupling of test code and infrastructure
● Hard to scale
● No sharing of resources between test jobs on CI
● Tests are bound to simulator host machine
● Too easy to mess with simulator internals
16

17. New Model
17

18. New model
Not bound
to macOS
18
Device Server
Agent
Rake / Parallel Cucumber
http
Test Test Test Test Test Test
http http http http http

19. New model
Device Server
ssh ssh ssh ssh
macOS,
Xcode, …
19
Agent
Rake / Parallel Cucumber
Not bound
to macOS
http
Test Test Test Test Test Test
http http http http http
ssh ssh

20. New model
● Separation of concerns
● Well-defined interface
● Cheaper test environments
● Easy sharing of resources between test jobs on CI
20

21. Why move? Sharing
t Agent 1 Simulators Agent 2 Simulators
1
2
3
4
5
6
t Shared Simulators
1
2
3
4
21
before after

22. More reasons to move
● Black boxing
● Easier updates
● No more xcrun simctl, fbsimctl, idevice*
● No more messing with internals
22

23. ● Did not fit well with existing code
● We need custom actions
● We need flexibility and fast prototyping
Why not Selenium Grid?
23

24. Device Server
24

25. Tools we were using before
● xcrun simctl
● fbsimctl
● WebDriverAgent
● ideviceinstaller
25

26. Tools we use now
● fbsimctl
● WebDriverAgent
26

27. fbsimctl as a server
$ fbsimctl boot -- listen --http 9999 -- shutdown
listen started: Http: 9999 Hid: No stdin: false
GET: /5C34FF6A-4585-406A-A14F-4C0881B74110/list
$ curl http://localhost:9999/B98A0678-3724-4BF5-9A4C-E84E06DDA1DD/list
{"status":"success","subject":[{"event_type":"discrete","timestamp":1523370322,"subje
ct":{"arch":"x86_64","state":"Shutdown","model":"iPhone 6","name":"iPhone
6","os":"iOS
11.2","udid":"B98A0678-3724-4BF5-9A4C-E84E06DDA1DD"},"event_name":"list"}],"events":[
]}
27

28. fbsimctl over http – exposed actions
● approve
● clearKeychain
● config
● diagnosticQuery
● diagnostic
● hid
● install
● launch
● list
● listApps
● open
● record
● relaunch
● search
● setLocation
● tap
● terminate
● uninstall
● upload
● screenshot
28

29. ● boot
● create
● delete
● erase
● shutdown
● launch_xctest
● list_device_sets
fbsimctl over http – missing actions
29

30. ● boot
● create
● delete
● erase
● shutdown
● launch_xctest
● list_device_sets
fbsimctl over http – missing actions
30

31. ● Prepare and reset simulators on demand
● Start WebDriverAgent on simulators
● Hide away satisfying desired requirements
● Extra actions
● Pre-boot/Pool simulators
Why we need something on top of fbsimctl?
31

32. Remote Device API (Subset)
class RemoteDevice
...
def reset(timeout: 60); end
def clear_safari_cookies; end
...
def app_installed?(bundle_id); end
def create_driver(caps); end
...
def video_acquire; end
def video_start; end
def video_stop; end
def last_crash_log; end
32

33. Device Server model
33

34. Device Server model
34

35. Results & Next Steps
35

36. Results
● 1.5 hours → 0.5 hours
● 30 → 60 machines hosting simulators
● Decoupled tests and infrastructure
● Eased updates
● Simplified test runner
36

37. Next Steps
● Open-source
● Remove ssh
● Pre-booted simulators
37

38. Know-Hows
38

39. Simulators per host
MacBook Pro, Intel Core i7 @ 2.8 GHz, 4 Cores, 16 GB RAM
● 6 simulators
Mac Pro, Intel Xeon E5-2697 @ 2.7 GHz, 12 Cores, 128 GB RAM
● 3 VMs × 6 simulators
39

40. Resetting Simulator
Option 1: Straightforward
fbsimctl ${UDID} shutdown -- erase -- boot
Full boot ~ 18 seconds
40

41. Resetting Simulator
Option 2: Not so straightforward
cd $(fbsimctl list_device_sets)
fbsimctl ${UDID} shutdown
rm -rf ${UDID}
cp -rp ${UDID}_BACKUP ${UDID}
fbsimctl ${UDID} boot
Full boot ~ 8 seconds
41

42. $ fbsimctl B98A0678-3724-4BF5-9A4C-E84E06DDA1DD list_apps | grep MobileSafari
Bundle Name: MobileSafari | ID: com.apple.mobilesafari | Install Type system | Container
/Users/qa/Library/Developer/CoreSimulator/Devices/B98A0678-3724-4BF5-9A4C-E84E06DDA1DD/data/Containers/Data/
Application/22D18402-3D7B-4F28-88CF-98935328B0FB
$ rm -f /Users/qa/Library/Developer/CoreSimulator/Devices/B98A0678-3724-4BF5-9A4C-E84E06DDA1DD/data/
Containers/Data/Application/22D18402-3D7B-4F28-88CF-98935328B0FB/Library/Cookies/Cookies.binarycookies
Reset Cookies
42

43. Cleaning Up Disk
/private/var/folders/*/*/*/*-*-*/*.app
/private/var/folders/*/*/*/fbsimctl-*
$(fbsimctl list_device_sets)/*/data/Library/Caches/ ⤸
com.apple.mobile.installd.staging/*/*.app
43

44. Setting Up Simulator Host
# enable on-screen keyboard
defaults write com.apple.iphonesimulator ConnectHardwareKeyboard -bool false
# disable location
defaults write com.apple.iphonesimulator LocationMode "3101"
# turn off chrome
defaults write com.apple.iphonesimulator ShowChrome -bool false
# EnableKeyboardSync
# GraphicsQualityOverride
# OptimizeRenderingForWindowScale
# SlowMotionAnimation
44

45. Managing Infrastructure
● 64 hosts to manage + 6 TeamCity agents
● xcversion https://github.com/KrauseFx/xcode-install
● Ansible playbook to update fbsimctl, xcode, deploy configs, etc
● Ansible to remove outdated simulators after updates
45

46. Recap
Parallelisation
Separation
Know-Hows
46

47. Thank you!

48. 48

49. ● calabash-ios,
● WebDriverAgent
● FBSimulatorControl
● cucumber-ruby
● parallel_cucumber
49
Our automation stack

50. Remote Device API
class RemoteDevice
attr_reader :info
attr_reader :wda_endpoint
attr_reader :calabash_port
attr_reader :device_ref
def initialize(server_endpoint, device_ref); end
def release; end
def reset(timeout: 60); end
def await(timeout: 60); end
def set_location(lat:, lon:); end
def clear_safari_cookies; end
50

51. Remote Device API
def approve_access(bundle_ids); end
def app_installed?(bundle_id); end
def install_app(app_bundle_path); end
def launch_app(bundle_id); end
def list_apps; end
def open_url(url); end
def relaunch_app(bundle_id); end
def terminate_app(bundle_id); end
def uninstall_app(bundle_id); end
51

52. Remote Device API
def create_driver(caps); end
def user_ports; end
def endpoint_for(port); end
def screenshot(filetype, prefer_wda: true); end
def video_acquire; end
def video_start; end
def video_stop; end
def last_crash_log; end
end
52