Slides from the presentation "From GitOps to a scalable CI/CD Pattern for Kubernetes" at the Docker New York City meetup, by Andrew Phillips. See https://www.meetup.com/Docker-NewYorkCity/events/257539512/

1. From GitOps to an scalable
CI/CD pattern for Kubernetes
Andrew Phillips
Docker New York City; Jan 16, 2019

2. From GitOps to an scalable
CI/CD pattern...for everything?
Andrew Phillips
Docker New York City; Jan 16, 2019

3. The bio slide
● Been on most sides of this space: developer, infra builder, product owner,
evangelist and more
● Long-standing open-source contributor
● Author and regular conference and meetup presenter
● Co-organizer of ContainerDays Boston & NYC

4. Agenda
1. What’s the challenge?
2. CI/CD practices & patterns today
3. What does Kubernetes add into the mix?
4. A CI/CD pipeline for Kubernetes
5. Next steps for your scenario
6. Q&A

5. The context
● Lots of organizations looking at Kubernetes right now
● Trying to use this also as an opportunity to “clean up” sub-optimal software
delivery pipelines
● How to do this right?

6. The challenge
● Lots of new runtime-specific concepts to deal with
● “Kubernetes-native” best practices still very much in flux
● Wrapping your head around “new” practices (e.g. GitOps) is hard
● Extrapolating from new practices in theory to actual, working implementations
is even harder
● Figuring out how all this new stuff relates to accepted industry practices and
tools is harder still

7. Failure mode: CCAOA
Cargo Cult All Over Again

8. The approach
● Recall good practices that we’d like to retain
● Incorporate refinements related to Kubernetes to create a straw man setup
● Develop a mental model to understand advantages and shortfalls of the setup
● Refine the setup based on tradeoffs to be made related to each user’s
particular situation
● Implement using appropriate tools

9. Existing good practice
● Reproducible builds
● Store source and derived artifacts appropriately
● Minimize duplication, especially around environment config
● Keep the business process flexible and the env automation robust
● Support 4 related processes:
○ Application update
○ Environment (config) update
○ Environment spin-up/restore
○ Environment drift detection and remediation
● Specifically, support env-specific (e.g. log settings change) and cross-env
(e.g. new app version rollout) processes concisely

10. Deployment execution: business process vs. technical process
Release pipeline A business process, represented as a sequence, possibly very specific to a service
Test environment Staging environment
Technical components with
interdependencies, defined
“as-code”. To be
automatically sequenced if
possible
Production environment
Deploy to Test Review Approve Deploy to Prod...
App
Endpoint
Config
App
Endpoint
Config’
Old app version
Namespace

11. What’s new with Kubernetes
● Declarative == intent/desired state is front-of-mind
● “as-code” description of the intended state of an environment
(“environments-as-code”)
● Purpose of automation is to modify that intent and apply it to the target
runtime(s)
● Needed UX (generally poor currently!) is to see/understand current intent;
review and approve changes to intent; and track status of intent (“being
applied”, “successfully applied”, “failed”, “drifted” etc.)

12. Deployment execution: adding in Kubernetes
Release pipeline A business process, represented as a sequence, possibly very specific to a service
Test environment Staging environment
Technical components with
interdependencies, defined
“as-code”. To be
automatically sequenced if
possible
Production environment
“environments-as-code” repositories
Deploy to Test Review Approve Deploy to Prod...
App
Endpoint
Config
App
Endpoint
Config’
Old app version
Namespace

13. TL;DR
Release pipeline A business process, represented as a sequence, possibly very specific to a service
Test environment Staging environment
Technical components with
interdependencies, defined
“as-code”. To be
automatically sequenced if
possible
Production environment
“environments-as-code” repositories
Deploy to Test Review Approve Deploy to Prod...
App
Endpoint
Config
App
Endpoint
Config’
Old app version
Namespace
Imperative pipeline across environments
Declarative spec for each environment
(with support for some imperative “cheating” where necessary)

14. What’s popular in the Kubernetes community
● Actuation based on reconciliation engine built into runtime with continuous
enforcement
○ Interest in “pull-based”, async invocation via repo-watching
● Out-of-the-box support for some types of rollout via Deployment object, as
well as CRDs to define your own
○ But can also manipulate underlying objects directly
● GitOps ~ environments-as-code + async invocation + repo workflow for
business process

15. The straw man

16. ● App repo is driving deployment, env repo is snapshotting desired cluster state
○ Think source code in github is to container image in registry as template in app repo is to
manifest in env repo
● Env repo is just a checkpoint in time, cluster can evolve
○ Kubernetes applies changes to manifests
○ Strategies like exponential rollouts or traffic shifting apply changes over time
● Env repo is not a guaranteed healthy state
○ We can defer snapshotting until some health/success metric… but rollback has no silver bullet
● Deletion from either repo ≠ deletion from cluster
○ Challenge is: not trivial to know if cluster depends, or will depend on manifest not submitted to
repo
○ kubectl apply --prune attempts to solve this with a lot of (scary) caveats
Mental model

17. One team, one env, app pipeline only

18. team1-repo
env-repo
applied-prod-branch
master-
branch
service
config
deployment

19. Multiple envs

20. env-repo
applied-staging-branch applied-prod-branch
team1-repo
master-
branch
service
config
deployment
+ overrides
Values that need to change
across in variables: could be
specified as patches, or
variables, or...

21. Updating settings for a single env only

22. The mental model
● Understand how your app is updated across two dimensions:
● code change promoted through to
prod
● common externalized config setting,
e.g. localized title ● rotating database credentials for
prod
● updating discovery service
endpoint
● adding debug logging to staging
● configuring a scaling policy in prod
based on a new metric
dependent on app version
independent
of
environment
dependent
on
environment
independent of app version

23. The mental model
● Understand how your app is updated across two dimensions:
Application update
Static environment config
update
(App-linked) environment config
update
dependent on app version
independent
of
environment
dependent
on
environment
independent of app version

24. The mental model
● Understand how your app is updated across two dimensions:
Application update
Static environment config update
(App-linked) environment config
update
dev responsibility ~ app repo platform responsibility ~ env repo

25. team1-repo
env-repo
applied-staging-branch applied-prod-branch
master-
branch
service
config
+
overrides
staging-
branch
deployme
nt
prod-
branch
deployme
nt
Settings can only be
updated by running
the full app pipeline,
requiring an app
rebuild. Advantage is
no config duplication
Settings in the env
can be updated
without rebuilding the
app. Downside is
more config
duplication.

26. Multiple teams

27. team1-repo
env-repo
applied-staging-branch applied-prod-branch
master-
branch
service
config
+
overrides
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
+
overrides
staging-
branch
config
prod-
branch
config

28. The four related processes
● Application update
○ “I want to validate a new release candidate and promote it through envs to prod”
● Environment (config) update
○ “I want to change the attributes of a particular env only”
● Environment spin-up/restore
○ “I want to (re-)create an environment from scratch, with config from a specific checkpoint/point
in time”
● Environment diff, drift detection and remediation
○ “I want to understand how the actual config of an env relates to the intended config, also
potentially across different environments”

29. App1 update

30. team1-repo
env-repo
applied-staging-branch applied-prod-branch
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-bra
nch
config
apiVersion: v1
kind: Service
metadata:
name: app1-service
...
spec:
type: ClusterIP
ports:
- port: 808080
targetPort: http
protocol: TCP
name: http
selector:
app: app1
env: {{ environment }}

31. team1-repo
env-repo
applied-staging-branch
app1.yaml
applied-prod-branch
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-bra
nch
config
apiVersion: v1
kind: Service
metadata:
name: app1-service
...
spec:
type: ClusterIP
ports:
- port: 808080
targetPort: http
protocol: TCP
name: http
selector:
app: app1
env: {{ environment }}
Step 1
apiVersion: v1
kind: Service
metadata:
name: app1-service
...
spec:
type: ClusterIP
ports:
- port: 8080
targetPort: http
protocol: TCP
name: http
selector:
app: app1
env: staging
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: app1
namespace: staging
...
---
apiVersion: v1
kind: ConfigMap
...

32. team1-repo
env-repo
applied-staging-branch
app1.yaml
applied-prod-branch
app1.yaml
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-bra
nch
config
apiVersion: v1
kind: Service
metadata:
name: app1-service
...
spec:
type: ClusterIP
ports:
- port: 808080
targetPort: http
protocol: TCP
name: http
selector:
app: app1
env: {{ environment }}
Step 2
apiVersion: v1
kind: Service
metadata:
name: app1-service
...
spec:
type: ClusterIP
ports:
- port: 8080
targetPort: http
protocol: TCP
name: http
selector:
app: app1
env: production
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: app1
namespace: production
...
---
apiVersion: v1
kind: ConfigMap
...

33. App2 update

34. team1-repo
env-repo
applied-staging-branch
app1.yaml
applied-prod-branch
app1.yaml
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-
branch
config
app.version: 0.0.34
port: 8090
image: gcr.io/image@sha256:{{
container.image.digest }}

35. team1-repo
env-repo
applied-staging-branch
app1.yaml
app2.yaml
app2-env.yaml
applied-prod-branch
app1.yaml
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-
branch
config
Step 1app.version: 0.0.34
port: 8090
image: gcr.io/image@sha256:{{
container.image.digest }}
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: app2
namespace: staging
...
spec:
replicas: 1
...
image:
gcr.io/image@sha256:abcdef
ports:
- name: http
containerPort: 90
---
...
---
apiVersion: v1
kind: ConfigMap
...
data:
config.yaml: |
'app.version': 0.0.4

36. team1-repo
env-repo
applied-staging-branch
app1.yaml
app2.yaml
app2-env.yaml
applied-prod-branch
app1.yaml
app2.yaml
app2-env.yaml
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-
branch
config
Step 2
app.version: 0.0.34
port: 8090
image: gcr.io/image@sha256:{{
container.image.digest }}
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: app2
namespace: production
...
spec:
replicas: 1
...
image:
gcr.io/image@sha256:abcdef
ports:
- name: http
containerPort: 90
---
...
---
apiVersion: v1
kind: ConfigMap
...
data:
config.yaml: |
'app.version': 0.0.4

37. App2 prod config change

38. team1-repo
env-repo
applied-staging-branch
app1.yaml
app2.yaml
app2-env.yaml
applied-prod-branch
app1.yaml
app2.yaml
app2-env.yaml
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-
branch
config
apiVersion: v1
kind: ConfigMap
metadata:
name: app2-env-config
namespace: production
labels:
app: app2
env: production
data:
config.yaml: |
'log.level': INFOTRACE

39. team1-repo
env-repo
applied-staging-branch
app1.yaml
app2.yaml
app2-env.yaml
applied-prod-branch
app1.yaml
app2.yaml
app2-env.yaml
master-
branch
service
config
staging-
branch
deployme
nt
prod-
branch
deployme
nt
team2-repo
master-
branch
custom-s
ettings
staging-
branch
config
prod-
branch
config
Step 1
apiVersion: v1
kind: ConfigMap
metadata:
name: app2-env-config
namespace: production
labels:
app: app2
env: production
data:
config.yaml: |
'log.level': INFOTRACE
apiVersion: v1
kind: ConfigMap
metadata:
name: app2-env-config
namespace: production
labels:
app: app2
env: production
data:
config.yaml: |
'log.level': TRACE

40. The four related processes
Demonstrated:
● Application update
● Environment (config) update
Should be relatively easy to see how to do the following with the same data:
● Environment spin-up/restore
● Environment diff, drift detection and remediation

41. Straw man in a nutshell
● Source config in app repo, “compiled” (a.k.a. hydrated) config in env repo(s)
○ Cf. source code in app repo, compiled code in artifact repository
● One app repo per app/team, shared env repos for environments
● Use branches where possible to represent different environments
○ If greater separation via access control is needed, use different repos or consider other
storage
● Keep config in app repo unless it needs to be independent of the app lifecycle
or you really, really need to be able to change it for one env
○ Allows you to use templating/overriding to cut down on duplication
○ The fewer places to keep track of config, the better
● Source config doesn’t have to be “raw” YAML, can be more suitable
abstraction!
○ Cf. higher-level language source code vs. low-level assembly code

42. A quick refresher on the overall picture...
Release pipeline A business process, represented as a sequence, possibly very specific to a service
Test environment Staging environment
Technical components with
interdependencies, defined
“as-code”. To be
automatically sequenced if
possible
Production environment
“environments-as-code” repositories
Deploy to Test Review Approve Deploy to Prod...
App
Endpoint
Config
App
Endpoint
Config’
Old app version
Namespace

43. Implementation thoughts
● Prefer explicit invocation over “repo-watching”
○ Avoids “root-level” process running inside cluster, and provides more flexibility for multi-step
rollouts
○ “Repo-watching” makes visualizing current status harder, and requires some sort of feedback
mechanism to distinguish successful from failed deployments in the repo
○ Hard to support phased/multi-step application
○ Easier to reproduce/simulate

44. Implementation thoughts
● Commit to master (also) after successful application, not (just) before
○ Avoids having to distinguish attempted from successful deployments on master
○ Allows for richer pre-application validation than code diff in a PR (e.g. three-way diff against
actual environment)
○ Avoids commit rights to repo being equivalent to deploy rights to env (and your automation will
need commit rights to make pull requests, unless you used forked repos)
○ Harder if pull requests are used for the business process (as in vanilla GitOps) - requires
multiple branches or “on approve” deployment

45. Tuning for your scenario
● Appropriate level of abstraction? How much “raw” Kubernetes YAML
should our developers have access to?
● Where should the abstraction live? In templates? In CRDs? In the
automation tool?
● What to use for templating? Token replacement or overrides?
● When do you snapshot/publish to the env repo? On every change to the
cluster? Or when a desired end-state is reached? (think multi-step rollout)
● Access control: how many repos do you need? Are code repositories right
for your use case, or e.g. a blobstore better?
● Am I distributing or deploying? Helm is much better suited for distribution
than deployment
○ Its templating capability is often used as part of deployment flows, though

46. Implement
● Choose storage implementations and partitioning strategies for your
environments-as-code
● Define the appropriate level of abstraction for your developers and choose
tools to support it
● Choose a flexible automation tool for your deployment business process
● Define an appropriate definition of deployment health/success to determine
when a deployment is “good”
● Decide which of the four processes - app update, env config update, env
restore and env drift detection - you want to support
● Build pipelines
● Done!

47. Thank you!