Join us for a webinar on securing the DevOps lifecycle with GitOps. Explore the best defenses for common security threats to code repositories, and see how to apply GitOps best practices to your CICD pipelines for Kubernetes. The adoption of GitOps already increases the security and stability of your Kubernetes deployment pipelines, keeping your deployment credentials and other secrets inside of the cluster. Although GitOps improves CICD pipeline security, it shifts the security burden to Git itself. For organizations who wish to defend themselves from malicious internal or external actors, or who operate under high compliance requirements, implementing additional security measures to Git provides identity guarantees, automation of change control, and detailed audit trails. In this webinar, we’ll discuss 4 common Git attacks and how to mitigate them: 1. User impersonation 2. Malicious user tampering with the repository’s history 3. Malicious user attacking the Git platform 4. Historical attacks on Git clients and their impact

1. Continuous
Security for GitOps
April 2019 Webinar
Brice Fernandes – brice@weave.works – @fractallambda
1

2. ● Building cloud-native OSS and commercial products since
2014 (Weave Net, Moby, Kubernetes, Prometheus)
● Founding member of CNCF
● Weave Cloud runs on Kubernetes since 2015
● We developed “GitOps” - more later!
● Kubernetes support subscriptions, training and consulting
2
About Weaveworks

3. Typical CICD pipeline
Continuous Integration
Cluster API
Continuous Delivery/Deployment
Container
Registry
CI
Code
Repo
Dev RW
CI credsGit creds
RW
CR creds3
RO
RW
API creds
CR creds1
Shares credentials cross several logical security boundaries.
Boundary
RO RW
Container
Registry (CR)
creds2

4. The GitOps Model
4

5. 5
GitOps is...
An operation model

6. 6
GitOps is...
An operation model
Derived from CS and operation knowledge

7. 7
GitOps is...
An operation model
Derived from CS and operation knowledge
Technology agnostic (name notwithstanding)

8. 8
GitOps is...
An operation model
Derived from CS and operation knowledge
Technology agnostic (name notwithstanding)
A set of principles (Why instead of How)

9. 9
GitOps is...
An operation model
Derived from CS and operation knowledge
Technology agnostic (name notwithstanding)
A set of principles (Why instead of How)
Although
Weaveworks
can help
with how

10. 10
GitOps is...
An operation model
Derived from CS and operation knowledge
Technology agnostic (name notwithstanding)
A set of principles (Why instead of How)
A way to speed up your team

11. 11
1 The entire system is described declaratively.

12. 12
1 The entire system is described declaratively.
Beyond code, data ⇒
Implementation independent
Easy to abstract in simple ways
Easy to validate for correctness
Easy to generate & manipulate from code

13. 13
The canonical desired system state is versioned
(with Git)
2

14. 14
The canonical desired system state is versioned
(with Git)
Canonical Source of Truth (DRY)
With declarative definition, trivialises rollbacks
Excellent security guarantees for auditing
Sophisticated approval processes (& existing workflows)
Great Software ↔ Human collaboration point
2

15. 15
Approved changes to the desired state are
automatically applied to the system
3

16. 16
Approved changes to the desired state are
automatically applied to the system
Significant velocity gains
Privileged operators don’t cross security boundaries
Separates What and How.
3

17. 17
Software agents ensure correctness
and alert on divergence
4

18. 18
Software agents ensure correctness
and alert on divergence
4
Continuously checking that desired state is met
System can self heal
Recovers from errors without intervention (PEBKAC)
It’s the control loop for your operations

19. 19
1 The entire system is described declaratively.
2 The canonical desired system state is versioned
(with Git)
3 Approved changes to the desired state are
automatically applied to the system
4 Software agents ensure correctness
and alert on divergence

20. 20
Canonical
source of truth
People
Software
Agents
Software
Agents

21. Cluster API
GitOps pipeline
Container
Registry
CI
Code
Repo
Dev RO
CR creds2
CI credsGit creds
RO
Deploy
CR creds3
RO
RW
Config repo
creds
CR creds1
Credentials are never shared across a logical security boundary.
RW RW
RW
Cluster API
creds
Can al re
s a s e
Config Repo

22. Cluster API
GitOps pipeline
Container
Registry
CI
Code
Repo
Dev RO
CR creds2
CI credsGit creds
RO
Deploy
CR creds3
RO
RW
Config repo
creds
CR creds1
Credentials are never shared across a logical security boundary.
RW RW
RW
Cluster API
creds
Operator RW Config Repo

23. Cluster API
GitOps pipeline
Container
Registry
CI
Code
Repo
Dev RO
CR creds2
CI credsGit creds
RO
Deploy
CR creds3
RO
RW
Config repo
creds
CR creds1
Credentials are never shared across a logical security boundary.
RW RW
RW
Cluster API
creds
Operator RW Config Repo
Pro s & co t t
en c e t

24. Cluster API
GitOps pipeline
Container
Registry
CI
Code
Repo
Dev RO
CR creds2
CI credsGit creds
RO
Deploy
CR creds3
RO
RW
Config repo
creds
CR creds1
Credentials are never shared across a logical security boundary.
RW RW
RW
Cluster API
creds
Operator RW Config Repo
Ex e t a di g
an t ut *

25. 25
GitO n
p a t

26. Secure your GitOps pipeline
26

27. Move from access to cluster to access to
repository.
...So how to secure your repository?
Moving the burden of security
27

28. Securing your repositories
28

29. 29
Thank you
Over to you Andy
brice@weave.works
@fractallambda

30. Hardening Git for GitOps
@sublimino and @controlplaneio

31. I’m:
- Andy
- Dev-like
- Sec-ish
- Ops-y

33. Everything as Code

34. Applications, infrastructure, and security
as code... are deployed as code, who
protects that code?

35. Who protects that code?

37. Why Git?

38. How Git Works
● Git operates as a merkle tree.
● SHA-1 was considered broken for security purposes and deprecated by NIST
in 2011
● The Shattered attack pads data to collide two SHA-1 hashes, which could be
used to overwrite existing trusted commits with malicious code
● Git v2.13.0 moved to a hardened SHA-1 implementation which isn't
vulnerable to this attack.
● (for a deep dive see Ian Miell’s excellent Learn Git the Hard Way)

39. Source: https://www.ecmweb.com/power-quality/data-center-outage-costs-continue-rise

40. What is Continuous Security?
● Infrastructure as Code
● Security as Code
● Continuous Delivery

42. Veracity and Provenance

43. Threat #1: Git Users
Can Impersonate
Each Other

44. https://github.com/jayphelps/git-blame-someone-else/commit/e5cfe
4bb2190a2ae406d5f0b8f49c32ac0f01cd7

45. Threat #1: Git Users Can Impersonate Each Other
● Mitigation: Enforce Strong Identity in VCS (GitHub/GitLab) with GPG Signed
Commits
● Physical GPG Keys increase security
● Mitigation: Run GPG-Validating Code in CI

46. Threat #2: Malicious
User Rewrites History

47. Threat #2: Malicious User Rewrites History
● When to use Force Push
● Mitigation: Prevent Force Pushes to Master Branch
● Mitigation: Backup Git Repositories

48. Threat #3: Malicious
User Removes
Security Features

49. Threat #3: Malicious User Removes Security
Features
● Mitigation: Configure Git Provider with Infrastructure as Code
● Mitigation: Monitor Git Provider’s Audit Logs
● Mitigation: Verify Commits to Master

50. Threat #4: Old Git
client versions are
insecure

51. Attacks on Git
Teleport Attacks
Branch Teleport Attack A branch merge point is moved to point to a “WIP” or to a buggy
code commit. This gets automerged on a developer’s pull.
Tag Teleport Attack A tag pointer is moved to a different place in the history and the
wrong version is retrieved. For example to a previous version
with known vulnerabilities.
Rollback Attacks
Branch Rollback Attack Critical code is omitted.
Global Rollback Attack Critical code is omitted.
Effort Duplication Attack Coding effort is increased.
Deletion Attacks
Branch Deletion Attack A branch is missing.
Tag Deletion Attack A tag is deleted.

52. Threat #4: Old Git Client Versions Are Insecure
● Mitigation: Keep Software Versions Updated

53. Further considerations

54. Where do I version control my secrets?
● Paper/USB/CDR and two fireproof safes?

55. Where do I version control my secrets?
● Paper/USB/CDR and two fireproof safes?
● Vault (or actually Consul)? https://www.vaultproject.io/docs/auth/kubernetes.html

56. Where do I version control my secrets?
● Paper/USB/CDR and two fireproof safes?
● Vault (or actually Consul)? https://www.vaultproject.io/docs/auth/kubernetes.html
● Sealed Secrets (a Kubernetes controller and tool for one-way encrypted Secrets):
https://github.com/bitnami-labs/sealed-secrets
● Git Crypt (GPG and Git integration): https://www.agwa.name/projects/git-crypt/

57. Further Considerations
● Restrict deployments with policy (working hours, prevent deployments to production on a
Friday afternoon, or enforce security teams to review certain changes)
● Permitting only dedicated reviewers to merge Pull Requests can act as a change control or
security review gate, with automatically generated release notes reflecting the application
changes that are about to be deployed and that provide a clear picture of the changes.
● Static analysis can also be performed on the contents of the Pull Requests themselves with
tools such as kubesec.io
● As part of a wider security strategy, tools like Notary, Grafeas, and in-toto prevent old
Docker images from being deployed, as do Kubernetes cluster admission
controllers such as Kritis and Porteiris.

58. Persisting Configuration:
Continuous Security

59. Continuous
Security

60. Thanks!
https://control-plane.io
@controlplaneio
https://www.weave.works/blog/whitepaper-hardening-git-for-gitops