"Secure development on Kubernetes" With the rise of Kubernetes, the Java developer has arrived in the DevOps age as well. By the multitude of complex tasks, the necessary security is often neglected. Even in managed clusters of well-known cloud providers, there are many traps and points of attack lurking. In this presentation, essential security-critical components of a Kubernetes cluster will be presented. Security problems and corresponding measures to mitigate these will be shown. All steps are described using live demos with an exemplary Spring Boot Java application, that is deployed as a docker container in a Kubernetes cluster, taking into account recommended security patterns. Speaker: Andreas Falk, Novatec Consulting Talk language: English About the Speaker: ********************* Andreas Falk has been working in enterprise application development projects for more than twenty years. Currently, he is working as a managing consultant for Novatec Consulting located in Germany. In various projects, he has since been around as consultant, architect, coach, developer, and tester. His focus is on the agile development of cloud-native enterprise java applications using the complete Spring platform. As a member of the Open Web Application Security Project (OWASP), he likes to have a closer look at all aspects of application security as well. Andreas is also a frequent speaker at conferences like Spring I/O, CloudFoundry Summit, Devoxx, and OWASP AppSec.

1. Secure Development On
Kubernetes
Security Meetup by SBAResearch
10.06.2020
Andreas Falk

2. andreas.falk@novatec-gmbh.de / @andifalk
https://www.novatec-gmbh.de/beratung/agile-security
Introduction
Andreas Falk
Novatec Consulting
2

3. Agenda
1. What Can Go Wrong?
2. Application Security
3. Container Security
4. Kubernetes Security
5. Kubernetes Secrets
3

4. Look here:
https://github.com/andifalk/secure-development-on-kubernetes
4
Where are the Slides and theCode?

5. Introduction
5
What can go wrong?

6. Severe Vulnerability in Kubernetes
Source: https://blog.aquasec.com
6

7. Crypto Mining Via K8sDashboard
Source: https://blog.heptio.com
7

8. Open ETCD Portsin Kubernetes (1)
https://shodan.io
8

9. Open ETCD Portsin Kubernetes (2)
9
$ etcdctl --endpoints=http://xx.xx.xx.xx:2379
cluster-health
member b97ee4034db41d17 is healthy: got healthy
result
from http://xx.xx.xx.xx:2379
cluster is healthy
https://github.com/etcd-io/etcd/releases

10. Vulnerable Docker Images
Source: The state of open source security report (snyk.io)
10

11. All is Root
11

12. Application- / Docker- /K8s-Security
12
Sowhat can WE do as
Developers?

13. The Path for Secure Development on K8s
Application
Security
Container
Security
Kubernetes
Security
Kubernetes
Secrets
13

14. The Path for Secure Development on K8s
Security
ApplicationContainer
Security
Kubernetes
Security
Kubernetes
Secrets
14

15. Application Security
15
WebApplication
Authentication
Authorization
SQLInjection
Cross Site Scripting (XSS)
Cross Site Request Forgery (CSRF)
Data Protection (Crypto)
...

16. Application Security
16

17. Iteration 1: Application Security
https://github.com/andifalk/secure-development-on-kubernetes
Live Demo: Show me thecode
17

18. ▪ Input Validation for ALL types of input
▪ Output Encoding to preventXSS
▪ Use only Parameterized Queries/Prepared Statements
▪ Enforce Authentication &Authorization
▪ Never implement your own Crypto orSession-Management
▪ Check 3rd Party Dependencies for Vulnerabilities
▪ Use Static & Dynamic Application SecurityTesting
18
Application Security 101
https://cheatsheetseries.owasp.org
https://owasp.org/www-project-top-ten
https://owasp.org/www-project-proactive-controls

19. The Path for Secure Development on K8s
Application
Security
Container
Security
Kubernetes
Security
Kubernetes
Secrets
19

20. OWASP DockerTop 10
20
1. Secure User Mapping
2. Patch Management Strategy
3. Network Segmentation and Firewalling
4. Secure Defaults and Hardening
5. Maintain Security Contexts
6. Protect Secrets
7. Resource Protection
8. Container Image Integrity and Origin
9. Follow Immutable Paradigm
10. Logging
https://github.com/OWASP/Docker-Security
https://doi.org/10.6028/NIST.SP.800-190
https://github.com/OWASP/Container-Security-Verification-Standard
https://www.bsi.bund.de

21. Virtual Machine (VM) Basics
Physical Hardware
Hypervisor (VMM)
Guest OS Guest OS
Guest Apps Guest Apps
Physical Hardware
Host OS
Guest OS Guest OS
Guest Apps Guest Apps
VMM
Type 1 Virtual MachineMonitor Type 2 Virtual MachineMonitor
ESXi
Workstation
21

22. Container (Security) Basics
Linux Namespaces
Control Groups (cgroups)
Capabilities
Mandatory Access Control
Secure Computing Mode
Secrets Management
Container
Linux Namespaces
Control Groups (cgroups)
Capabilities
Mandatory Access Control
Secure Computing Mode
Secrets Management
Container
LXD
Linux Host (LinuxKernel)
22

23. ▪ Process IDs
▪ Network
▪ Mount Points
▪ Inter-Process Communications (IPC)
▪ User & Group IDs
▪ Unix Timesharing System (UTS): hostname & domainnames
▪ Control groups (cgroups)
23
Linux Kernel Namespaces
$ man namespaces
$ sudo lsns

24. Linux Control Groups (cgroups)
▪ Resource Limits
− CPU
− Memory
− Devices
− Processes
− Network
24
For Java this only works with container aware
JDK versions as of OpenJDK 8u192 or above
Recommendation: Use Java 11

25. ▪ Break up privileges into smallerunits
− CAP_SYS_ADMIN
− CAP_NET_ADMIN
− CAP_NET_BIND_SERVICE
− CAP_CHOWN
− ...
25
Linux Capabilities
http://man7.org/linux/man-pages/man7/capabilities.7.html
$ man capabilities
$ docker run --cap-drop=ALL --cap-add=NET_BIND_SERVICE

26. ▪ Restrict System Calls
− Secure Computation Mode (seccomp)
− Google gVisor
26
▪ Linux Kernel Security Modules (MAC)
− AppArmor
− Security-Enhanced Linux (SELinux)
Linux Mandatory AccessControl & System Calls
https://docs.docker.com/engine/security/seccomp
https://apparmor.net
https://en.wikipedia.org/wiki/Security-Enhanced_Linux
https://gvisor.dev/docs

27. Docker Images
Linux Host
...
cgroups
namespaces
Base Image (Alpine, …)
Dockerfile
Application
Container Image
Base Image (Alpine, …)
Dockerfile
Application
Container Image
Container Registry
docker run
27

28. USERdirective in Dockerfile
Say No To Root (1)
FROM openjdk:11-jre-slim
COPY hello-spring-kubernetes-1.0.0-SNAPSHOT.jar app.jar
EXPOSE 8080
RUN addgroup --system --gid 1002 app && adduser
--system --uid 1002 --gid 1002 appuser
USER 1002
ENTRYPOINT java -jar /app.jar
https://opensource.com/article/18/3/just-say-no-root-containers
28

29. Use JIB and Distroless Images
29
Say No To Root (2)
https://github.com/GoogleContainerTools/jib
plugins {
id 'com.google.cloud.tools.jib' version '...'
}
jib {
container {
user = 1002
}
}

30. Container Image Security
Dockerfile
Application
Container Registry
Container Image
Base Image (Alpine, …)
Image Security
Scanner
Vulnerable 3rd party
dependencies
OSvulnerabilities
OSvulnerabilities
30
https://anchore.com/opensource/
https://github.com/coreos/clair
https://github.com/aquasecurity/trivy
https://www.docker.com/blog/announcing-scanning-from-snyk-for-docker

31. https://www.docker.com/press-release/Docker-Snyk-Announce-Partnership-Container-Vulnerability-Scanning
31

32. Iteration 2: Container Security
https://github.com/andifalk/secure-development-on-kubernetes
Live Demo: Show me thecode
32

33. ▪ Learn Linux (Security) Basics
▪ Load Images from Trusted RegistriesOnly
▪ Scan Images for Vulnerabilities (inCI/CD Pipeline)
▪ Say No To Root & Run with --security-opt=no-new-privileges
▪ Do NOT hardcode Secrets into a ContainerImage
▪ Limit resources (memory, CPU,processes, ...)
▪ Use Linux Security Module (seccomp, AppArmor,SELinux)
33
Container Security 101
https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html
https://docs.docker.com/engine/security
https://blog.aquasec.com/docker-security-best-practices
https://blog.aquasec.com/devsecops-with-trivy-github-actions

34. The Path for Secure Development on K8s
Application
Security Security
Security
Container Kubernetes Kubernetes
Secrets
34

35. Kubernetes Basics
Ingress Service Deployment Replica Set
Pod
Pod
Pod
https://kubernetes.io/docs/concepts
https://www.aquasec.com/wiki/display/containers/70+Best+Kube
rnetes+Tutorials
35

36. 36
Source:
Kubernetes Security, O’Reilly, 2018
Kubernetes attack vectors

37. Operational / Development Kubernetes Security
37
https://kubernetes.io/docs/concepts/security/overview/#the-4c-s-of-cloud-native-security
https://learnk8s.io/production-best-practices/
K8s Development Security
TLS
Auth
Authz
Master Node
API Server Scheduler
Etcd Controller
Manager
TLS
Auth
Authz
Worker Node
Kubelet Container
Runtime
Kube Proxy
K8s Operational Security

38. Kubernetes Security
Kubernetes Auditing
Network Policies
Role BasedAccess
Control (RBAC)
Resource Limits
Pod Security Context
Pod Security Policy
Open Policy Agent

39. Resource Limits
39
https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource
https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource
spec:
...
containers:
resources:
limits:
cpu: "1"
memory: "512Mi"
requests:
cpu: 500m
memory: "256Mi"
...

40. Pod/Container Security Context
spec:
securityContext:
runAsNonRoot: true
containers:
securityContext:
allowPrivilegeEscalation: false
privileged: false
runAsNonRoot: true
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
https://kubernetes.io/docs/tasks/configure-pod-container/security-context
40

41. Pod Security Policy (Still In Beta!)
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: no-root-policy
spec:
privileged: false
allowPrivilegeEscalation: false
requiredDropCapabilities:
- ALL
runAsUser:
rule: 'MustRunAsNonRoot'
...
https://kubernetes.io/docs/concepts/policy/pod-security-policy
41

42. Kubernetes Role Based AccessControl (RBAC)
ClusterRole ClusterRoleBinding
Role RoleBinding
SubjectAPI Groups
Resources
Verbs
Cluster-Wide
42
Namespace
- User
- Group
- ServiceAccount
https://kubernetes.io/docs/reference/access-authn-authz/rbac/

43. Kubernetes Role Based AccessControl (RBAC)
43
https://kubernetes.io/docs/reference/access-authn-authz/rbac/
apiGroups extensions, apps, policy, ...
resources pods, deployments, configmaps, secrets,nodes,
services, endpoints, podsecuritypolicies, ...
verbs get, list, watch, create, update, patch, delete, use,...

44. Open Policy Agent
44
https://www.openpolicyagent.org
https://play.openpolicyagent.org

45. Open Policy Agent -Kubernetes Gatekeeper
45
https://github.com/open-policy-agent/gatekeeper

46. Helm 3 Is Here!
46
https://v3.helm.sh
https://helm.sh/docs/faq/#removal-of-tiller

47. Iteration 3: Kubernetes Security
https://github.com/andifalk/secure-development-on-kubernetes
Live Demo: Show me thecode
47

48. ▪ Follow Container Security 101
▪ Use a Managed Kubernetes Cluster
▪ Enable Audit Logs
▪ Enforce Authentication & Role Based Access Control
▪ Use Pod Security Policies / Open Policy Agent
▪ Upgrade to Helm Version 3.x(Remove Tiller)
▪ Monitor your Kubernetes Cluster
48
Kubernetes Security 101
https://cheatsheetseries.owasp.org
https://owasp.org/www-project-top-ten
https://owasp.org/www-project-proactive-controls

49. The Path for Secure Development on K8s
Application
Security
Container
Security Security
Kubernetes
Kubernetes
Secrets
49

50. Kubernetes Secrets
Secrets
KMS
Secrets
Secrets
Etcd

51. Kubernetes Secrets
51
https://kubernetes.io/docs/concepts/configuration/secret
apiVersion: v1
kind: Secret
metadata:
name: hello-spring-cloud-kubernetes
namespace: default
type: Opaque
data:
user.username: dXNlcg==
user.password: azhzX3VzZXI=
admin.username: YWRtaW4=
admin.password: azhzX2FkbWlu

52. ▪ Encrypt Secret Data at Rest & inTransit
− Only Base64 encoded by default inEtcd!
▪ Restrict interactions with secretsAPI (RBAC)
▪ Mount secrets instead of ENV Mapping
52
Kubernetes Secrets - BestPractices
https://kubernetes.io/docs/concepts/configuration/secret/#best-practices
https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data

53. Encryption Layers
53

54. Envelope Encryption On Kubernetes
54
https://cloud.google.com/kms/docs/envelope-encryption
https://kubernetes.io/docs/tasks/administer-cluster/kms-provider

55. Key Management System (KMS)Providers
▪ Azure Key Vault
▪ Google Cloud KMS
▪ AWS KMS
▪ Hashicorp Vault
▪ ...
https://github.com/Azure/kubernetes-kms
https://github.com/Azure/kubernetes-keyvault-flexvol
https://cloud.google.com/kms
https://aws.amazon.com/de/kms
https://learn.hashicorp.com/vault/kubernetes/external-vault
55

56. What about Secretsin
56
▪ Sealed Secrets
▪ Helm Secrets
▪ Kamus
▪ Sops
▪ Hashicorp Vault
https://learnk8s.io/kubernetes-secrets-in-git
https://github.com/bitnami-labs/sealed-secrets
https://github.com/futuresimple/helm-secrets
https://github.com/Soluto/kamus
https://github.com/mozilla/sops
https://www.vaultproject.io

57. Summary
57

58. ▪ Kubernetes is Complex - CheckAlternatives!
▪ Follow Application Security 101
▪ Follow Container & Kubernetes Security 101
▪ Ensure your secrets are encrypted inK8s
▪ Never store secrets in Source Control(Git, …)
▪ Check out the Demos:
https://github.com/andifalk/secure-development-on-kubernetes
58
Summary / KeyInsights

59. Books and OnlineReferences
59

60. ▪ Kubernetes Security, O’Reilly, 2018, ISBN: 978-1-492-04600-4
▪ Container Security, O’Reilly, 2020, ISBN: 978-1492056706
▪ https://github.com/andifalk/secure-development-on-kubernetes
▪ Crafty Requests: Deep Dive Into Kubernetes CVE-2018-1002105 - Ian Coldwater(Video)
▪ Ship of Fools: Shoring Up Kubernetes Security - Ian Coldwater(Video)
▪ https://kubernetes.io/docs/concepts/security/overview/#the-4c-s-of-cloud-native-security
▪ https://kubernetes.io/docs/tasks/administer-cluster/securing-a-cluster
▪ https://opensource.com/article/18/3/just-say-no-root-containers
▪ https://github.com/GoogleContainerTools/jib
▪ https://anchore.com/opensource/
▪ https://github.com/coreos/clair
▪ https://github.com/aquasecurity/trivy
▪ https://www.owasp.org/index.php/OWASP_Docker_Top_10
60
Books and Online References(1)

61. ▪ https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource
▪ https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource
▪ https://kubernetes.io/docs/tasks/configure-pod-container/security-context
▪ https://kubernetes.io/docs/concepts/policy/pod-security-policy
▪ https://kubernetes.io/docs/reference/access-authn-authz/rbac/
▪ https://kubernetes.io/docs/concepts/configuration/secret
▪ https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data
▪ https://cloud.google.com/kms/docs/envelope-encryption
▪ https://kubernetes.io/docs/tasks/administer-cluster/kms-provider
▪ https://github.com/Azure/kubernetes-kms
▪ https://cloud.google.com/kms
▪ https://aws.amazon.com/de/kms
61
Books and Online References(2)

62. Andreas Falk
62
Managing Consultant
Mobil: +49 151 46146778
E-Mail: andreas.falk@novatec-gmbh.de
Novatec Consulting GmbH
Dieselstraße 18/1
D-70771 Leinfelden-Echterdingen
T. +49 711 22040-700
info@novatec-gmbh.de
www.novatec-gmbh.de