Usually, companies have a clear separation between testing environments and development environments. But what if we take a different approach and combine everything into one environment and still survive? The speaker will discuss ways to organize cloud environments for testing and development, compare them, and discuss the pros and cons of the combined approach.

1. Cocktail of Environments
How to Mix Test and Development Environments and Stay Alive

2. @aatarasoff
@aatarasoff
@aatarasoff
2
@aatarasoff

3. 3

4. Prologue: No Good Solutions
4

5. 5
No microservices - no problems

6. 6
No microservices - no problems
Testing locally - easy

7. 7
No microservices - no problems
Testing locally - easy
Testing in dev
environment - easy

8. 8
No microservices - no problems
Testing locally - easy
Testing in dev
environment - easy
Testing in test/stage
environment - easy

9. 9
No microservices - no problems
Testing in dev
environment - easy
Testing in test/stage
environment - easy
Testing locally - easy

11. Testing one - easy

12. Testing one - easy
Testing many - hard

13. Chapter 1: Baking Dev Environment
13

14. 14
Work on my machine

15. 15
Work on my machine
Limited number of
services

16. 16
Work on my machine
Limited number of
services
Separated
configuration

17. 17
Telepresence

18. 18
Telepresence
Async processes?

19. 19
Local Async Process Testing

20. 20
Local Async Process Testing
Who will read
a message?

21. Benefits
● Fast feedback loop
● Good for simple usecases
Drawbacks
● Hard to test complex scenarios
● Hard to collaborate
○ QA
○ other developers
Local Only
21

22. 22
Full Copy for each Developer

23. 23
Full Copy for each Developer

24. 24
Full Copy for each Developer
Own logical DB

25. 25
Full Copy for each Developer
Own logical DB
Own topics and
subscriptions

26. Issues to
address
● Tool to establish “Vasya” env
○ 10-15 minutes max
● Handle the load
○ 10k workloads and much more
● Support separated infra components
○ dbs in containers
○ emulators for queues
26

27. Issues to
address
● Tool to establish “Vasya” env
○ 10-15 minutes max
● Handle the load
○ 10k workloads and much more
● Support separated infra components
○ dbs in containers
○ emulators for queues
27

28. Issues to
address
● Tool to establish “Vasya” env
○ 10-15 minutes max
● Handle the load
○ 10k workloads and much more
● Support separated infra components
○ dbs in containers
○ emulators for queues
28

29. Optimizations
● Part copy instead of full
○ core services first
○ specific services on-demand
● Shutdown every night
● Env per squad not developer
● Get rid of X
○ do not use service mesh
○ do not keep logs
29

30. Optimizations
● Part copy instead of full
○ core services first
○ specific services on-demand
● Shutdown every night
● Env per squad not developer
● Get rid of X
○ do not use service mesh
○ do not keep logs
30

31. Optimizations
● Part copy instead of full
○ core services first
○ specific services on-demand
● Shutdown every night
● Env per squad, not developer
● Get rid of X
○ do not use service mesh
○ do not keep logs
31

32. Optimizations
● Part copy instead of full
○ core services first
○ specific services on-demand
● Shutdown every night
● Env per squad, not developer
● Get rid of X
○ do not use service mesh
○ do not keep logs
32

33. Benefits
● Full isolation
● The cognitive load is low
Drawbacks
● Custom configuration
● High resources consumption
● You own - you troubleshooting
Full Copy for each Developer
33

34. 34
Service Injection

35. 35
Service Injection
As a developer I created new
branch: feature/mp-101-bla-bla

36. 36
Service Injection
Deploy each branch to
dev cluster

37. 37
Service Injection
x-service-route: payment-service:mp-101

38. 38
Service Injection
x-service-route: payment-service:mp-101
Nginx unpacks cookie to header

39. 39
We Need More Branches
x-service-route: payment-service:mp-101::cart-service:mp-102

40. 40
We Need More Branches
Async processes?

41. Benefits
● Low resources consumption
● Less troubleshooting required
● Convinient collaboration
Drawbacks
● Shared resources - poor isolation
● Hard to test async processes
Service Injection
41

42. Chapter 2: What are You, Stable Dev?
42

43. 43
Typical Environments

44. 44
Atypical Environments

45. 45
Atypical Environments

46. 46
One Cluster - Several Environments

47. 47
Stable Dev
Always contains all the services with
the same versions as in production

48. 48
Stable Dev
Default routes come to it
Always contains all the services with
the same versions as in production

49. 49
Stable Dev
Foundation for developing, testing, and staging

50. 50
Stable Dev
Foundation for developing, testing, and staging
Steady as a rock

51. 51
Stable Dev
Foundation for developing, testing, and staging
Steady as a rock
Developers should
not TEST on it

52. 52
Branch Dev
Developers test
in branch

53. 53
Branch Dev
“x-service-route” Cookie or HTTP Header

54. 54
Canary Dev
Every new release
is deployed as a
candidate first

55. 55
Canary Dev
Every new release
is deployed as a
candidate first
x-service-route: payment-service:rc

56. 56

57. What if not?
● No dogfooding
○ A silo between QA and developers
○ Developers are pushed to fix the stage
● We should keep stable two environments instead of one
○ Staging should be stable by design
○ The development environment should be stable too
■ If the authorization service doesn’t work -
developer cannot test their branch
57

58. What if not?
● No dogfooding
○ A silo between QA and developers
○ Developers are pushed to fix the stage
● We should keep stable two environments instead of one
○ Staging should be stable by design
○ The development environment should be stable too
■ If the authorization service doesn’t work -
developer cannot test their branch
58

59. Chapter 3: Make Some Code
59

60. Traffic Routing
60

61. 61
Service Injection
x-service-route: payment-service:mp-101

62. 62
Service Injection
x-service-route: payment-service:mp-101
Nginx unpacks cookie to header

63. Istio Virtual Service
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: payment-service
spec:
...
http:
- name: stable
route:
- destination:
host: payment-service.services.svc.cluster.local
63

64. Istio Virtual Service
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: payment-service
spec:
...
http:
- name: stable
route:
- destination:
host: payment-service.services.svc.cluster.local
64
Deploy for every stable version
via Helm chart

65. Route to a Branch
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: payment-service
spec:
...
http:
- name: payment-service-mp-101
match:
- headers:
x-service-route:
regex: ^(payment-service:mp-101.*|.*::payment-service:mp-101.*)$
- name: stable
route:
- destination:
host: payment-service.services.svc.cluster.local 65
We cannot add it with
Helm chart

66. Virtual Service Merge Operator
---
apiVersion: istiomerger.monime.sl/v1alpha1
kind: VirtualServiceMerge
metadata:
name: payment-service-mp-101
spec:
patch:
http:
- name: payment-service-mp-101
match:
- headers:
x-service-route:
regex: ^(payment-service:mp-101.*|.*::payment-service:mp-101.*)$
target:
name: payment-service
66
Deploy for every branch
via Helm chart

67. 67
Propagation Problem

68. 68
Propagation Problem
x-service-route should be propagated

69. 69
Propagation Problem
x-service-route should be propagated
Tracing is a MUST!

70. 70
Propagation Problem
x-service-route should be propagated
Tracing is a MUST!
x-b3-trace-id in response

71. 71
Tricky Case: Migrations that Break
You want to test
migrations in your branch

72. 72
Tricky Case: Migrations that Break
However, you may
affect Stable Dev
You want to test
migrations in your branch

73. 73
Solution: Migrations that Break

74. 74
Solution: Migrations that Break
We chose this variant

75. 75
Tricky Case: Webhooks

76. 76
Tricky Case: Webhooks
Nobody knows about
your internal
infrastructure

77. 77
Solution #1: Webhooks
We can switch URL
for webhook on the
third-party side

78. 78
Solution #1: Webhooks
We can switch URL
for webhook on the
third-party side
However, we affect
Stable Dev

79. 79
Solution #2: Webhooks
Create advanced mock or Fake
to get rid of
External Service dependency at all

80. 80
Solution #3: Webhooks
Use Smart-Proxy
and some correlation ID
for matching requests
from the external service

81. 81
Solution #3: Webhooks
We chose this approach
Use Smart-Proxy
and some correlation ID
for matching requests
from the external service

82. Unblocking Async Scenarios
82

83. 83
Async Issues

84. 84
Async Issues
Who should
process a message?

85. 85
Let’s Use the Same Appoach

86. 86
Let’s Use the Same Appoach
Developers can
interfere with one
another

87. 87
Subscription per Branch

88. Issues to
address
● Static subscription for canary
● Dynamic subscriptions for branches
● Common library
○ context propagation
○ message skip logic
88

89. Issues to
address
● Static subscription for canary
● Dynamic subscriptions for branches
● Common library
○ context propagation
○ message skip logic
89

90. 90
Service Catalog

91. 91
Service Catalog
Metadata such as
name, team,
domain, resources

92. 92
Service Catalog
On metadata change
the state machine
tries to re-apply

93. 93
Service Catalog
Modules have
unified interface

94. 94
Service Catalog
Modules use Terraform for
resource management

95. 95
Service Catalog
Ot they can also perform
additional tasks such as
auto-filling the Vault
configuration for a service

96. 96
Deployment Process

97. 97
Deployment Process
Developers can change
meta.yml for their service in
Service Catalog repository

98. 98
Deployment Process
New meta data will be applied

99. 99
Deployment Process
The state machine will align
resources with new meta

100. 100
Deployment Process
Application Delivery Pipeline
checks the state

101. 101
Static Subscriptions
Re-run the state
machine
Change the
Pub/Sub module

102. 102
Static Subscriptions
Re-run the state
machine
Change the
Pub/Sub module
some-subscription-rc
some-subscription-qa

103. Issues to
address
● Static subscription for canary
● Dynamic subscriptions for branches
● Common library
○ context propagation
○ message skip logic
103

104. 104
Dynamic Subscriptions

105. 105
Dynamic Subscriptions
Register on branch
first pipeline

106. 106
Dynamic Subscriptions
Register on branch
first pipeline
Re-apply Pub/Sub
module

107. 107
Dynamic Subscriptions
Register on branch
first pipeline
Re-apply Pub/Sub
module
some-subscription-mp-101

108. 108
Dynamic Subscriptions
Deregister on
branch deletion

109. 109
Dynamic Subscriptions
Deregister on
branch deletion
Re-apply Pub/Sub
module

110. 110
Dynamic Subscriptions
Deregister on
branch deletion
Re-apply Pub/Sub
module
order-subscription-mp-101

111. 111
Deployment Process

112. Issues to
address
● Static subscription for canary
● Dynamic subscriptions for branches
● Common library
○ context propagation
○ message skip logic
112

113. 113
Common Library

114. 114
Common Library
Skip or Process?

115. 115
Common Library: Decision Maker
It is NOT FOR me

116. 116
Common Library: Decision Maker
It is NOT FOR me It is NOT FOR me

117. 117
Common Library: Decision Maker
It is NOT FOR me It is NOT FOR me
It is FOR me

118. 118
Common Library: Decision Maker
It is NOT FOR me It is NOT FOR me
It is FOR me
?

119. 119
Common Library: Decision Maker
Should be aware of all
other versions

120. 120
Common Library
Skip or Process?
Put it back to the
client context

121. 121
Complex Scenarios Supported

122. Issues to
Address
● Separated DB for branches
○ the same approach as for subscriptions
122

123. 123
Separated DBs Schema

124. 124
Separated DBs Schema

125. Issues to
Address
● Separated DB for branches
○ the same approach as for subscriptions
○ the incomplete data
125

126. Chapter 4: Ephemeral Environments
126

127. 127
Welcome to Real Life

128. 128
Welcome to Ephemeral Environments

129. Types of
Ephemeral
Environments
● One service branch
● Several services branches
○ under one x-source-route
○ Jira-based
● Custom environments
○ for squad (payment-dev)
○ for domain (warehouse)
129

130. Types of
Ephemeral
Environments
● One service branch
● Several services branches
○ under one x-source-route
○ Jira-based
● Custom environments
○ for squad (payment-dev)
○ for domain (warehouse)
130

131. 131
Custom Ephemeral Environments

132. Epilogue: Some Conclusions
132

133. Benefits
● No silo between Dev and QA
● Low resources consumption
● Environments on-demand
133

134. Drawbacks
● High cognitive load
● Time investments
● Not-fair isolation
134

135. @aatarasoff
@aatarasoff
@aatarasoff
Questions?
@aatarasoff
135