Slides for Cookpad Tech Kitchen (English version) http://cookpad-tech-kitchen.connpass.com/event/37908/

1. Cookpad Tech Kitchen #2
~ continuous delivery for iOS ~

2. Cookpad(domestic)
2

3. Cookpad(domestic)
3
2016年12月期 第2四半期
https://cf.cpcdn.com/info/assets/wp-content/uploads/20160809153218/2016.2Qir.pdf

4. Cookpad(world)
4

5. Cookpad(world)
5
2016年12月期 第2四半期
https://cf.cpcdn.com/info/assets/wp-content/uploads/20160809153218/2016.2Qir.pdf

6. my.name // Kazuaki Matsuo
my.role // Test Engineer
my. // @Kazu_cocoa
my. // KazuCocoa
my. // kazucocoa.wordpress.com
let my = Profile()
6

8. Agenda
✓About our development style
✓ relationship between
departments
✓ Dev process / flow
✓Ice break
✓ Test strategy
✓ Test tools / tips
✓Testing strategy and what
test we run
✓ UI Testing
8

9. Our relationship
~ departments ~
9

10. Relationship(2014)
mobile first department
Mobile app engineersService engineers
10

11. Relationship(2015)
mobile infrastructure
11
Mobile app engineersService engineers

12. Relationship(2016)
mobile infrastructure
12
Mobile app engineersService engineers

13. Our relationship
~ roles in iOS development ~
13

14. Owner
test engineers
designers
directors
iOS
engineers
Our roles(2016)
14

15. Our dev cycle
Our dev process
15

16. previous
release
Deadline of
Pull Request
Code
Freeze
submit release
about a month
5 days2 weeks 3 days 4 days
Dev cycle
🍎🍏
16

17. Dev process
🍎
design/implement/check
🍏
17
submit release

18. Today’s focus point
18

19. Today’s focus point
Dev Flow
~ PR ~
Pact for iOS
UI Testing
19

20. Today’s focus point
UI Testing
20

21. Today’s focus point
Pact for iOS
21

22. Today’s focus point
Dev Flow
~ PR ~
22

23. What I talked
23
✓ About Cookpad
✓ Our relationship between departments
✓ Our role in iOS development
✓ Our development cycle and process
✓ Who is going to proceed talking each
topics

24. My role and work
~ Focus on technical activities ~
24

25. ~ pyramid ~
25
Our test strategy

26. Small
Medium
Large
Enormous
manual
Our test strategy
26
http://techlife.cookpad.com/entry/2016/08/13/test-size-for-mobile

27. Small
Medium
Large
Enormous
http://techlife.cookpad.com/entry/2016/08/13/test-size-for-mobile
Our test strategy
27
manual

28. ✓ Automated tests
✓ running tests, judgement the results
✓ GUI Testing
✓ HTTP Requests check
✓ …
✓ Manual tests
✓ new features
✓ difficult to run automatically
✓ exploratory testing
What we check
in test size L/E
28

29. ✓ Automated tests
✓ running tests, judgement the results
✓ GUI Testing
✓ HTTP Requests check
✓ …
✓ Manual tests
✓ new features
✓ difficult to run automatically
✓ exploratory testing
What we check
in test size L/E
29

30. ~ GUI Test / Scenario Test ~
~ L/E size ~
UI Testing
30

31. ✓ Run tests based on scenarios
✓ We cover over 80% screen transitions
✓ Frequency we run tests
✓ arbitrary timing or before release the
app
Test cases
31

32. ✓ Scenario based automated test
✓ Capture HTTP requests
✓ Screen captures
✓ Image diff
Our automated tests
32

33. ✓ Unexpected broken scenario
✓ Unexpected broken network traffic
✓ Unexpected broken layout
✓ Crashes
What we can find out
33

34. Unexpected broken
network traffic
34

35. Unexpected broken
network traffic
35

36. Example of traffics
36

37. only for iOS8.2
Unexpected broken
layouts
Order of Dialog Button
is revered
37

38. only for iOS8.2
Unexpected broken
layouts
Here is diff
38

39. Expected layouts
except for iOS8.2
This order is correct
39

40. ✓ Easy to figure out what kind of
unexpected defects caused
✓ Regression test
✓ Humans can do other creative activities,
other tasks and so on while running the
automated tests
Advantage of
automated tests
40

41. ✓ If we run all tests with manual
✓ Scenario/GUI diff
✓ It takes around a few hours per a
device at least
✓ We run several devices per release
✓ We can reduce human resources to do
this kind of tasks
Costs
41

42. iOS 8.0 8.1 8.2 8.3 8.4 9.0 9.1 9.2 9.3 10.0
iPho
ne
◯ ◯ ◯ ◯ ◯ ◯ ◯ ◯ -
iPad ◯ ◯ ◯ ◯ ◯ ◯ ◯ ◯ -
iPad
Pro
◯ ◯ ◯ ◯ -
✓ ◯: select devices we run tests
✓ 6~8 devices per a release
✓ iOS8.0 was removed from Xcode
✓ Do test partially with manual
Combination
42

43. http://www.slideshare.net/KazuMatsu/20141018-selenium-appiumcookpad
Architecture
Test scenario / wrapper / frameworks
43

44. ✓ We should update scenario every
release
✓ BTW, other libraries and wrappers
are updated only to solve unstable
environment
✓ We can update scenario within a day
✓ Include libraries and wrappers
✓ We can do other tasks except for
automated test
Maintenance costs
44

45. ✓ Currently, we try convert
Objective-C to Swift
Code changes
45

46. ✓ Keeping 99.9% crash free users
Crashes(Answer)
46

47. ✓ What is our motivation to enhance
test automation
✓ 2014/1~: I joined to Cookpad
✓ I’m first test engineer
✓Test is not only my activity
✓We’d like to ensure minimum quality
to release the application
Motivation
47

48. ~ our work ~
UI Testing
48

49. ✓ include libraries and scenarios
✓ Appium 0.18.0 ~>
How long have we
developed this tests?
49

50. ✓ We should define concrete plans to
tests
✓ What kind of test should we plan?
✓ We should consider software design,
configuration and so on against
software tests
✓ I used Mindmap to arrange test targets
How to design tests
50

51. Maindmap
51

52. Example of mind maps
52

53. ~ tool stack for iOS ~
UI Testing
53

54. ✓ UI Testing
✓ Appium (JavaScript)
✓ Turnip/RSpec (Ruby)
✓ EarlGrey (Objective-C/Swift)
✓ WebDriverAgent(Objective-C)
✓ image diff
✓ kobold (JavaScript)
✓ Others
✓ wiremock (Jave)
✓ http_proxy (Elixir)
Tool stack for iOS
54

55. ✓ UI Testing
✓ Appium (JavaScript)
✓ Turnip/RSpec (Ruby)
✓ EarlGrey (Objective-C/Swift)
✓ WebDriverAgent(Objective-C)
✓ image diff
✓ kobold (JavaScript)
✓ Others
✓ wiremock (Jave)
✓ http_proxy (Elixir)
Tool stack for iOS
55

56. Appium
56

57. ✓ Backend differ from:
✓ Appium 1.6.x
✓ WebDriverAgent for xcuitest strategy
✓ UIAutomator for appium strategy
✓ Before Appium 1.5.x
✓ UIAutomator for appium strategy
✓ Appium try to solve problems between
iOS <=> Appium Server
✓ Users only know about interface
against Appium server
Appium
57

58. EarlGrey
58

59. ✓ XCUITest supports over iOS9
✓ Cookpad supports iOS8.0
✓ Lifecycle of XCUITest is same as
XCTest
✓ So, we can’t clean environment
every tests.
✓ e.g. Permission dialog
✓ This make tests flaky.
✓ BTW, we should do XCUITest based UI
Testing in the future.
How about XCUITest ?
59

60. Others
✓ image diff
✓ kobold (JavaScript)
✓ Others
✓ wiremock (Jave)
✓ http_proxy (Elixir)
60

61. End
61