Slides of talk given at London Study of Enterprise Agile Meetup in June 2019. We go over GitOps and how it affects delivery speed in software development and release.

1. Speeding up your
team with GitOps
London SEAM – June 2019
Brice Fernandes – brice@weave.works
1

2. 2
I’m Brice
I work for Weaveworks.
You can find Weaveworks at https://www.weave.works
or @weaveworks
Team at Weaveworks is behind the GitOps model.
We build GitOps tools for enterprise Kubernetes and
Cloud Native.

3. 3
This talk
Tactical
Technical
Process / tools

4. 4
This talk
Tactical
Technical
Process / tools
Principles of Operations

5. What is GitOps?
5

6. 6
GitOps is...

7. 7
GitOps is...
An operation model for

8. 8
GitOps is...
An operation model
Derived from CS and operation knowledge

9. 9
GitOps is...
An operation model
Derived from CS and operation knowledge
Technology agnostic (name notwithstanding)

10. 10
GitOps is...
An operation model
Derived from CS and operation knowledge
Technology agnostic (name notwithstanding)
A set of principles (Why instead of How)

11. 11
GitOps is...
An operation model
Derived from CS and operation knowledge
Technology agnostic (name notwithstanding)
A set of principles (Why instead of How)
Although
Weaveworks
can help
with how

12. 12
GitOps is...
An operation model
Derived from CS and operation knowledge
Technology agnostic (name notwithstanding)
A set of principles (Why instead of How)
A way to speed up your team

13. The GitOps Model
13

14. 14
Kubernetes
Cluster
GitOps ON Kubernetes

15. 15
GitOps ON Kubernetes
Kubectl /
Direct access

16. 16
GitOps ON Kubernetes

17. 17
Configuration
Repository
GitOps ON Kubernetes

18. 18
GitOps ON Kubernetes

19. 19
Deployment
Agent *
GitOps ON Kubernetes

20. 20
Security
Boundary
*

21. 21
GitOps ON Kubernetes

22. 22
GitOps ON Kubernetes
Image
Repository

23. 23
GitOps ON Kubernetes
State
continuously
monitored

24. 24
GitOps ON Kubernetes
Control Loop

25. The Principles of GitOps
25

26. 26
1 The entire system is described declaratively.
2 The desired system state is versioned
3 Approved changes to the desired state are
automatically applied to the system
4 Software agents ensure correctness
and alert on divergence

27. 27
GitOps ON Kubernetes
Image
Repository
1
The entire system is described declaratively.

28. 28
GitOps ON Kubernetes
Image
Repository
2
The desired system state is versioned

29. 29
GitOps ON Kubernetes
Image
Repository
3
Approved changes to the desired state are
automatically applied to the system

30. 30
GitOps ON Kubernetes
Image
Repository
4
Software agents ensure correctness
and alert on divergence

31. What should be GitOps’ed?
31

32. What should be GitOps’ed?
32
I’m so very
sorry

33. 33
?

34. Dashboards
Alerts
Playbook
Kubernetes Manifests
Application configuration
Provisioning scripts
34
Application checklists
Recording Rules
Sealed Secrets

35. Dashboards
Alerts
Playbook
Kubernetes Manifests
Application configuration
Provisioning scripts
35
Application checklists
Recording Rules
Sealed Secrets
Technical / Application Domain

36. 36

37. Why should we care?
37

38. Typical CICD pipeline
Continuous Integration
Cluster API
Continuous Delivery/Deployment
Container
Registry
CI
Code
Repo
Dev RW
CI credsGit creds
RW
CR creds3
RO
RW
API creds
CR creds1
Shares credentials cross several logical security boundaries.
Boundary
RO RW
Container
Registry (CR)
creds2

39. Cluster API
GitOps pipeline
Container
Registry
CI
Code
Repo
Dev RO
CR creds2
CI credsGit creds
RO
Deploy
CR creds3
RO
RW
Config repo
creds
CR creds1
Credentials are never shared across a logical security boundary.
RW RW
RW
Cluster API
creds
Canonical desired
state store
Config Repo

40. Cluster API
GitOps pipeline
Container
Registry
CI
Code
Repo
Dev RO
CR creds2
CI credsGit creds
RO
Deploy
CR creds3
RO
RW
Config repo
creds
CR creds1
Credentials are never shared across a logical security boundary.
RW RW
RW
Cluster API
creds
Operator RW Config Repo

41. Operator
Cluster API
GitOps pipeline
Container
Registry
CI
Code
Repo
Dev RO
CR creds2
CI credsGit creds
RO
Deploy
CR creds3
RO
RW
Config repo
creds
CR creds1
Credentials are never shared across a logical security boundary.
RW RW
RW
Cluster API
creds
RW Config Repo
Process & constraints
enforcement

42. Operator
Cluster API
GitOps pipeline
Container
Registry
CI
Code
Repo
Dev RO
CR creds2
CI credsGit creds
RO
Deploy
CR creds3
RO
RW
Config repo
creds
CR creds1
Credentials are never shared across a logical security boundary.
RW RW
RW
Cluster API
creds
RW Config Repo
Exceptional auditing
and attribution*

43. 43
● Trivialises rollbacks
● Exceptional auditing and attribution*
● Separation of concerns
● No crossing security boundary
● Process & constraints enforcement
● Great Software ↔ Human collaboration point
● Easy to validate for correctness (Policies)
● System can self heal
Why should we care?

44. 44
● Trivialises rollbacks
● Exceptional auditing and attribution*
● Separation of concerns
● No crossing security boundary
● Process & constraints enforcement
● Great Software ↔ Human collaboration point
● Easy to validate for correctness (Policies)
● System can self heal
Why should we care?
* If you’ve secured your Git repositories properly

45. Gitops at Qordoba
45

46. 46
1) Estimated time needed to fix
prod software bugs ~60% less
time after switching
2) Estimated time to respond to
customer requests ~43% less
time after switching
3) Service uptime went to 100%
from 99%
Customer tools: Jenkins, GKE and
Weave Cloud. Apps: web, ML.
The GitOps effect

47. 47
The GitOps effect
~2 → 150+
deployments per week

48. 48
The GitOps effect
~2 → 150+
deployments per week
Why?

49. 49
Decide
Act
Observe
Orient

50. 50
Declare
Implement
Monitor /
Observe
Modify

51. 51
Declare
Implement
Monitor /
Observe
Modify

52. 52
Declare
ImplementModify
Continuous
Deployment
Default
dashboards
Automated by
software
agents
Monitor /
Observe

53. 53
Declare
ImplementModify
Continuous
Deployment
Default
dashboards
Automated by
software
agents
Monitor /
Observe
Software
making
commits

54. 54
Declare
ImplementModify
Continuous
Deployment
Default
dashboards
Automated by
software
agents
Monitor /
Observe
Safe and
reversible
changes

55. 55
Declare
ImplementModify
Continuous
Deployment
Default
dashboards
Automated by
software
agents
Monitor /
Observe
Automated,
templated
dashboards

56. 56
Feedback loop latency.
This is what matters.

57. Beyond technology
57

58. 58

59. BI dashboards
BI Reports
Playbook
Team Membership
Org Chart
Operations Manual
59
Security Policies
Processes
Roles & Authorisation

60. BI dashboards
BI Reports
Playbook
Team Membership
Org Chart
Operations Manual
60
Security Policies
Processes
Roles & Authorisation
Business Domain

61. 61
Hard
Technical
Problems

62. 62
Hard
Technical
Problems

63. 63
Hard
Human
Problems

64. 64
What if you could roll back your sales processes?
What if you could create a pull request on your
organisation policy?
What if you could get it reviewed and approved in an
afternoon?
What if your entire business could be reproduced the
same way a git repository is cloned?

65. GitOps at Weaveworks
65

66. 66
Kubernetes operator (Flux, Open Source)
Multiple clusters (staging and prod)
CD into staging
Promotion from staging to prod
Kubernetes
Automated diff tools
(*diff operators, Open Source)
Dashboard definitions in Git
(Grafanalib, Open Source)
Alert definitions in git
Read-only access to production
for all developers
Gated, PR-driven changes to
production
*“stress-reduced”

67. 67
Deploying a
service with
Flux

68. 68
Kubernetes operator (Flux, Open Source)
Multiple clusters (staging and prod)
CD into staging
Promotion from staging to prod
Kubernetes
Automated diff tools
(*diff operators, Open Source)
Dashboard definitions in Git
(Grafanalib, Open Source)
Alert definitions in git
Read-only access to production
for all developers
Gated, PR-driven changes to
production
*“stress-reduced”

69. 69

70. 70
Kubernetes operator (Flux, Open Source)
Multiple clusters (staging and prod)
CD into staging
Promotion from staging to prod
Kubernetes
Automated diff tools
(*diff operators, Open Source)
Dashboard definitions in Git
(Grafanalib, Open Source)
Alert definitions in git
Read-only access to production
for all developers
Gated, PR-driven changes to
production
*“stress-reduced”

71. 71

72. 72

73. 73
Kubernetes operator (Flux, Open Source)
Multiple clusters (staging and prod)
CD into staging
Promotion from staging to prod
Kubernetes
Automated diff tools
(*diff operators, Open Source)
Dashboard definitions in Git
(Grafanalib, Open Source)
Alert definitions in git
Read-only access to production
for all developers
Gated, PR-driven changes to
production
*“stress-reduced”

74. 74

75. 75

76. 76

77. 77

78. 78

79. 79

80. 80
Kubernetes operator (Flux, Open Source)
Multiple clusters (staging and prod)
CD into staging
Promotion from staging to prod
Kubernetes
Automated diff tools
(*diff operators, Open Source)
Dashboard definitions in Git
(Grafanalib, Open Source)
Alert definitions in git
Read-only access to production
for all developers
Gated, PR-driven changes to
production
⇒
*“stress-reduced”

81. 81
Kubernetes operator (Flux, Open Source)
Multiple clusters (staging and prod)
CD into staging
Promotion from staging to prod
Kubernetes
Automated diff tools
(*diff operators, Open Source)
Dashboard definitions in Git
(Grafanalib, Open Source)
Alert definitions in git
Read-only access to production
for all developers
Gated, PR-driven changes to
production
⇒
< 30 minute total disaster recovery

82. 82
Kubernetes operator (Flux, Open Source)
Multiple clusters (staging and prod)
CD into staging
Promotion from staging to prod
Kubernetes
Automated diff tools
(*diff operators, Open Source)
Dashboard definitions in Git
(Grafanalib, Open Source)
Alert definitions in git
Read-only access to production
for all developers
Gated, PR-driven changes to
production
⇒
< 30 minute total disaster recovery
Dozens of changes per day
with a very small team

83. 83
Kubernetes operator (Flux, Open Source)
Multiple clusters (staging and prod)
CD into staging
Promotion from staging to prod
Kubernetes
Automated diff tools
(*diff operators, Open Source)
Dashboard definitions in Git
(Grafanalib, Open Source)
Alert definitions in git
Read-only access to production
for all developers
Gated, PR-driven changes to
production
⇒
< 30 minute total disaster recovery
Dozens of changes per day
with a very small team
Incredibly fast
regression response

84. 84
Kubernetes operator (Flux, Open Source)
Multiple clusters (staging and prod)
CD into staging
Promotion from staging to prod
Kubernetes
Automated diff tools
(*diff operators, Open Source)
Dashboard definitions in Git
(Grafanalib, Open Source)
Alert definitions in git
Read-only access to production
for all developers
Gated, PR-driven changes to
production
⇒
< 30 minute total disaster recovery
Dozens of changes per day
with a very small team
Incredibly fast
regression response
Permissive approach
to production access

85. 85
Kubernetes operator (Flux, Open Source)
Multiple clusters (staging and prod)
CD into staging
Promotion from staging to prod
Kubernetes
Automated diff tools
(*diff operators, Open Source)
Dashboard definitions in Git
(Grafanalib, Open Source)
Alert definitions in git
Read-only access to production
for all developers
Gated, PR-driven changes to
production
⇒
< 30 minute total disaster recovery
Dozens of changes per day
with a very small team
Incredibly fast
regression response
Permissive approach
to production access
Excellent developer experience

86. 86
Kubernetes operator (Flux, Open Source)
Multiple clusters (staging and prod)
CD into staging
Promotion from staging to prod
Kubernetes
Automated diff tools
(*diff operators, Open Source)
Dashboard definitions in Git
(Grafanalib, Open Source)
Alert definitions in git
Read-only access to production
for all developers
Gated, PR-driven changes to
production
⇒
< 30 minute total disaster recovery
Dozens of changes per day
with a very small team
Incredibly fast
regression response
Permissive approach
to production access
Excellent developer experience
Stress-free on-call*

87. 87
Kubernetes operator (Flux, Open Source)
Multiple clusters (staging and prod)
CD into staging
Promotion from staging to prod
Kubernetes
Automated diff tools
(*diff operators, Open Source)
Dashboard definitions in Git
(Grafanalib, Open Source)
Alert definitions in git
Read-only access to production
for all developers
Gated, PR-driven changes to
production
⇒
< 30 minute total disaster recovery
Dozens of changes per day
with a very small team
Incredibly fast
regression response
Permissive approach
to production access
Excellent developer experience
Stress-free on-call*
*“stress-reduced”

88. Where to find out more
88
Search for “Weaveworks GitOps” in your favourite search engine
Take a look at our opensource work on https://github.com/weaveworks
Questions?
Weaveworks
@weaveworks
https://weave.works
Brice Fernandes
@fractallambda
brice@weave.works