GitOps provides a way for organizations to adopt continuous delivery practices using cloud native technologies like Kubernetes and Docker. It advocates storing all infrastructure configurations and code in a Git repository to enable automated deployment and management of applications. Weaveworks adopted GitOps and was able to recover from a complete system failure in just 45 minutes by having their entire system defined as code. GitOps follows the principles of continuous delivery and observability by integrating deployment pipelines, monitoring, and security policies to drive faster, more reliable software releases.

1. More, Better, Faster!
Cloud Native Apps with GitOps
Alexis Richardson
CEO, Weaveworks
& TOC Chair, CNCF
September 27, 2017

2. WTF
I want to show you how Weaveworks learned to
apply technology to make our business go much
faster, using cloud native and continuous delivery

3. WTF
GitOps – empowering developers to do operations

4. How did we get here

5. Remember this?

6. 1994

7. 1995

8. 1996

9. 2003

10. 2004

11. 2005

12. (1999-) 2008

13. Netflix wanted to improve streaming service for all, fast
Must Read!https://www.slideshare.net/AmazonWebServices/dmg206

14. 2009-2010 – cloud native appears as a set of requirements
https://www.slideshare.net/AmazonWebServices/dmg206
Must Read!

15. Investors pile in because: velocity equals big bucks

16. We are all developers now
● If you can merge a pull
request on GitHub then you
are a developer
● The industry has learnt
how to connect GitHub to
running applications via
CICD pipelines, enabling
Git to drive Ops… GitOps

17. Frontend
(Nginx)
The Interwebs
RDS
Instance
Frontend
(Nginx)
Frontend
(Nginx)
Authenticatin
g Proxy
WeaverangCollection
Service
AWS
DynamoDB
Query
Service
WeaverangControl
Service
AWS
SQS
WeaverangPipe Service
AWS
S3
Consul
Cluster
NATS
Cluster
Static
Content
Users
Service
Core Services
Scope
Services
AWS
DynamoDB
AWS S3
WeaverangDistribution
Service
WeaverangIngester
Service
Consul
Cluster
Cortex ServicesMonitoring & Management Services
Grafana
Prometheus
Weave
Scope
WeaverangNode
Exporter
WeaverangScope
Probes
Kubediff
One time, when we blew up Weave Cloud…

18. Before After

19. 45 minutes
from complete and total wipeout
back to happy

20. “I want to go faster”
Then
Adopt continuous delivery & align business and tech to work as one team
Tech teams are empowered to act quickly upon business needs, so they must be
multi-skilled and “own the system, own the changes”.
 more operations roles in “dev” teams
And
Use the right Cloud Native technologies
Automation means getting fewer errors and scaling safely
 Git, CICD, containers, orchestration
GitOps = Cloud Native + Continuous Delivery

21. Continuous Delivery

22. Fundamental Theorem of DevOps
What can be
described,
can be
automated
and accelerated

23. But most businesses release infrequently

24. DevOps while
ubiquitously
known has not
yet delivered
magic velocity
& continuous
everything

25. 1. New cloud Native apps
and tools  is forcing
change
2. Accelerate all the things
3. Automation phase shift –
adapting to many releases
per day
Summer is coming

26. Cloud Native

27. “Curate and promote tools for cloud native”
Microservices
Containerised apps
Dynamically orchestrated

28. Industry has bought in

29. Kubernetes is a platform for cloud native apps
“orchestration”
 Means that it runs containerised apps the way Linux runs processes
Powerful but ”low level” – will have many simplifying tools in future
Important:
It is declarative automated infrastructure
Kubernetes

30. We can store Kubernetes config in Git and validate it

31. The key to
velocity

32. Weave Kubernetes gRPC Docker Fluentd OpenTracing Prometheus
*** CUNNING SUBLIMINAL ADVERT ***

33. GitOps

34. • We use declarative infrastructure ie.
Kubernetes, Docker, Terraform, & more
• Our entire system including code, config,
monitoring rules, dashboards, is
described in GitHub with full audit trail
• We can roll our major or minor changes
as pull requests, and automatically
check for diffs if system diverges from
the desired “source of truth” in Git
How did Weaveworks rebuild our systems in 45 mins?

35. • Config is code
• Code must be version controlled
• Config must be version controlled too
GitOps follows the Logic of DevOps

36. GitOps follows the Logic of DevOps
• Config is code
• Code must be version controlled
• Config must be version controlled too
• What can be described can be automated
• Describe everything: code, config,
monitoring & policy; and then keep it in
version control

37. GitOps
• Git as a source of truth for desired state of whole system
• Compare desired with actual state to fire diff alerts
• Make ops changes by pull request

38. What this gets us
• Any developer can use GitHub
• Anyone can join team and ship a new
app or make changes easily
• All changes can be triggered, stored,
audited and validated in Git
And we didn’t have to do anything very
new or clever 

39. The future is joined up
• DevOps is evolving to accommodate
the potential of cloud native tools to
get more joined up CICD and release
automation at a much higher quality
• GitOps shows us how to join up
workflows and action oriented
dashboards in ways that make sense
for developers doing more ops

40. GitOps Deep Dive

41. GitOps journey
• Day 0 – push first app on first cluster & validate that it works
• Day 1 – add CICD updates & rollbacks via Git PRs
• Day 2 – observing and controlling a production system
• Day 3 – scale up – eg. better service routing (mesh) & security policy

42. The three pillars of GitOps
CompletePipeline
Observability
Security

43. Pipelines – ABCDE pattern
Deployment
App Dev Build (CI) Containers
Any Cluster
Any Cloud
Execution
Push
app to
cloud

44. Pipelines – ABCDE pattern
Deployment
App Dev Build (CI) Containers
Any Cluster
Any Cloud
Execution
Push app to
cloud

45. GitOps - do CD right
• Config is code & everything is config (‘declarative infra’)
• Code (& config!) must be version controlled
• CD tools that do not record changes in version
control are harmful

46. Continuous Delivery/Deployment
The GitOps Pipeline – automate releases, sync with Git
Image
Repo
OrchestratorDeploy
Synchronizer
Config change
Manual deployment
Git
Code change
Git
Update Hint
Continuous Integration
Deploy
Automator
CI
Pipeline

47. Takeaways
• Pushing apps & changes is the fundamental operation
• GitOps needs complete pipelines that join up CI, CD and
Release Automation in one flow
• The right tools must be used – they coordinate between
Git, CI, and the services running in the cluster, enabling
sophisticated deployment policies

48. Observability

49. GitOps & Observability
• If a change is released and no-
one is around to see it, then did
it really work?

50. Read the whole thing –
https://twitter.com/mipsytipsy/status/911711540008628224

51. Observability – understanding whole system wellness
• In GitOps we want to get
developers comfortable with
operational concepts like
monitoring, tracing, and
incident handling
• Like doctors, we must be able
to validate health as well as
diagnose problems, using a
common language and a
coherent set of tools

52. Observability – understanding whole system wellness
monitor
log
trace
visualize
troubleshoot
debugdiffs
alerts validate
Incident management & response

53. BigCos pioneer integrated solutions to spur faster action

54. Observability
Asking hard
questions of the
system is not the
same as setting up
a load of alerts

55. Diffs & auto sync are really great
Bake in metrics end to end and full stack from the start
For alerts, use RED metrics  focus on services
You can’t avoid some instrumentation – but that’s ok since all in Git
Visuals in Git – grafanalib
Policy & Rules in Git (traffic, incident management)
Automate (autogenerate) per-service screens & keep in Git
Some lessons we learnt running Weave Cloud

56. Digression

57. • Observability is a way to verify
that our system is in the desired
state as specified in Git eg. diffs
& alerts & more
• An observable system is one
that can be controlled, via a
feedback control loop that drives
continuous improvement
A bit of theory

62. The GitOps Pipeline is really driving a CONTROL LOOP…
GitOps
loop
Deployment
App Dev Build (CI) Containers
Any Cluster
Any Cloud
Execution
Release
ObserveOperate

63. Runtime
System
Monitoring
Tracing
Logging…
Control
System
Git Diff
People
Release Observe
Operate

64. System Actual
State
Observable
State
Understanding
Desired State
(in Git)
Observe
Orient
Decide
Act
Release
Diff
ROODA Loop

65. Fundamental Theorem of GitOps
What can be
described and
observed
can be
automated and
controlled and
accelerated

66. Takeaways
● Observability is fundamental to automation and understanding
● It is holistic and encompasses any question you could ask about the
difference between desired and observed state
● You must bake it in from start, using monitoring, tracing, diff tools …

67. Security (& Policy)

68. Recap…
• Day 0 – push first app on first cluster & validate that it works
• Day 1 – add CICD updates & rollbacks via Git PRs
• Day 2 – observing and controlling a production system
• Day 3 – scale up – eg. better service routing (mesh) & security policy

69. Who sees what
Who talks to whom
 Matters more as you scale
Based on rules
Routing, Firewalls, ACLs, Rollouts
 Declarative? Store them in Git
Security

70. ● By using diffs, we can immediately and automatically enforce
convergence to a correct (desired) system state
● SOX: Git repos control which developers touch the system, which via
GitOps CICD tooling can be mapped directly into running clusters
● Secrets?
Security: some examples we have seen @ Weaveworks

71. Parting Thoughts

72. ● A much easier way to deliver and manage better apps, faster
● Works anywhere!
● Much more resilient – 45 mins to recover from total system wipeout
What we got

73. ● Git is a source of truth *for everything* in cloud native era
● GitOps ROODA loop improves velocity & collaboration
● Focus on the 3 pillars: pipelines, observability & security central
This is leading us to new insights, new tools, new dashboards today
Key takeaways

74. Observe
Orient
Decide
Act
Release
GitOps is social
Understand
Team
Control
System

75. Why GitOps
The need for speed!
Business expects tech to be super responsive  consolidation of dev & ops skills in the most agile teams
Automation: a phase shift is coming
If we want to go from 1 release per MONTH to 1 or more release per DAY then we need to automate the
complete lifecycle
New app types will accelerate change
DevOps and cloud adoption have arrived. New application types are emerging.
Many use tools like Kubernetes & Docker which support “everything as code” and practices that deliver a
complete automated & accelerated lifecycle

76. Where ops happens
Powered by Kubernetes