the latest version of iOS 3x3

1. Keqiu Hu
Speed Up iOS Releases

2. 2
whoami
Software Engineer @ LinkedIn
Flagship infrastructure team

3. 3
Agenda
Background
Challenges
Q & A

4. 4
Background
Old days
One centralized mobile team
~15 iOS engineers
1 mobile release per month

5. Past:
One centralized mobile team
~15 iOS engineers
< 1 mobile release per month
5
Background

6. 6
Background
Now
Decentralized mobile team
150 iOS engineers
2000 commits/month
3x3

7. 7
3x3?
Release 3 times a day
Commit to publish time < 3 hours

8. 8
3x3?
Release 3 times a day
Commit to publish time < 3 hours

9. 9
3x3 Challenges
Reliability
Speed
Stability

10. Reliability

11. 11
Test Coverage

12. Layout Tests
Stress test views with different strings
12
Test Coverage

13. Scenario Tests
Replace manual testing
13
Test Coverage

14. Scenario Tests
Replace manual testing
14
Test Coverage

15. Push beta build to company
employees for internal testing
15
Dogfood

16. Speed

17. 17
Test Automation
7 days to 19 hours (5 hour compilation + 14 hours testing)

18. Speed up Swift compiling
(19 hours -> 15 hours)
18
Compiler

19. Speed up Swift compiling
19
Compiler

20. Speed up iOS testing framework
(15 hours -> 4 hours)
20
Test Framework

21. Distributed building and testing
(4 hours to 1 hour)
21
Distributed build!

22. Stability

23. Precommit -> Code base -> Post
commit
23
Stability
Pre-commit
Code
base
Post-
commit

24. 24
Stability
Testing infrastructure stability
Tooling stability

25. Hermetic testing environment
25
Environment Stability

26. 26
Test Stability
99.9%1000 = ?
99% 95% 90% 80% 50%

27. 27
Test Stability
99.9%1000 = 36.7%
99% 95% 90% 80% 50%

28. 28
Trunk Guardian
Flaky tests are worse than no tests
Disable all flaky tests

29. Synchronization between steps
29
Test Framework Stability

30. Synchronization between steps
30
Test Framework Stability

31. 31
Distributed Testing Stability
Running tests on 10 machines
Each node has a reliability of 95% and 95%^10 = 60%

32. 32
Distributed Testing Stability
Running tests on 10 machines
Each node has a reliability of 95% and 95%^10 = 60%
Capacity requirement

33. 33
Distributed Testing Stability
Running tests on 10 machines
Each node has a reliability of 95% and 95%^10 = 60%
Capacity requirement

34. Based on facebook’s xctool
Python wrapper
34
Project Hydra

35. Facebook’s xctool was declared EOL
No access to simulator APIs
35
Project Hydra

36. The project name is inspired by the
Matrix’s Bluepill
36
Project Bluepill

37. Objective-c code base on top of
CoreSimulator and other Apple
frameworks
Running tests in paralle via multiple
simulators
Automatically packing tests into groups
with similar running time
Retrying when simulator hangs or
crashes.
37
Project Bluepill

38. Open sourced!
github.com/linkedin/bluepill
38
Project Bluepill

39. Write a config:
{
"app": "./Sample.app", # Relative path or abs path
"scheme-path": "./SampleAppTestScheme.xcscheme", #
Relative path or abs path
"output-dir": "./build/" # Relative path or abs path
}
Run!
./bluepill –c config.json
39
Project Bluepill

40. Question?