Jay Beale We will attack a real Kubernetes cluster called Bust-a-Kube, which was released in 2019 as a free learning tool. The demonstration will start by compromising a real application running in a Kubernetes pod's container, gaining low privileged remote code execution inside that container. Next, we will explore what that compromised container can see on the cluster, finding the boundaries of its privileges. We will move laterally from that container to attack microservices on the cluster, gaining remote code execution in other containers, with higher privilege. We'll find that one of those can interfere with a final highest-privilege container. That highest privilege container will permit us to abuse the Kubernetes API to compromise the entire cluster. This demonstration will involve graphic "flags," allowing attendees to repeat the attack afterward as a downloadable solitaire "capture the flag" game. We'll then discuss and perform a second demo to teach defenses, working backward to defeat necessary steps in the first demo's chain of attacks. We'll demonstrate using pod security policies to force an AppArmor profile onto any pod (container) being deployed. We'll show how volume whitelists can block an attack, then demonstrate an evasion that defeats this defense. We'll then weaken this attack with root capability limits and AppArmor. We'll demonstrate an attack path where a bad actor can use a low-privilege Kubernetes cluster compromise to abuse the cloud provider APIs. This, in turn, leads to compromising the Kubernetes cluster more fully. We'll discuss how to break this attack using a cloud metadata API security feature that's Kubernetes-specific. In the course of these demonstrations, we'll conduct the attacks both manually and with an open source attack tool called Peirates. Finally, we'll discuss defenses that we did not use, including seccomp syscall whitelists, read-only root filesystems, and freely-available service meshes.

2. JAY BEALE
CTO, INGUARDIANS. (@JAYBEALE, @INGUARDIANS)
Kubernetes Practical
Attack and Defense

3. Graphical Bio

4. • API Server: Receives requests and and serves as first point of contact
• etcd Server: Stores the state of the cluster, alerts subscribed
components
• Controller Manager: Runs control loops to bring state to parity with
etcd
• Scheduler: Bin-packs containers onto nodes
• Kube-DNS: Gives every requested network endpoint a name
What Can We Attack on a Master Node?

5. • Kubelet: Ties the node back to the master components
• Container Runtime (e.g. Docker): Instruct the Linux kernel to create
containers
• Host Operating System
• Filesystem
• Network
• Kernel
• Workloads: Containers on the system
• Kube-Proxy: forwards traffic to each pod in a load-balanced network
service
What Can We Attack on a Worker Node?

6. Let's attack a Kubernetes
cluster, Scott Pilgrim-style!

7. In order to see Ramona
Flowers, Scott Pilgrim must
defeat her seven evil exes
The Story
Our attack demos will follow this path, with each new
access corresponding to battle with an evil ex.

8. Demo Time!
Scott Pilgrim vs the World
Evil Exes 1-4

9. • Pod Security Policies
• Volume Whitelists
• Root Capability Limits
• AppArmor
Defenses to Break the Attacks

10. Demo Time!
Breaking the Attacks

11. Demo Time!
Attacking
Evil Exes 5, 6 and 7

12. • Seccomp Syscall Whitelisting
• Read-only Root Filesystems
• Service Meshes
More Defenses to Break the Attacks

13. • Filtering system calls (syscalls) with seccomp has
two purposes:
• Restrict a compromised program's behavior to
the system calls in its profile
• Reduce the kernel’s attack surface
• Generate the syscall list with strace, then tell Docker
or Kubernetes to confine the pod to the known list.
Seccomp System Call Whitelisting

14. • Microservices lend themselves to this design
pattern
• Shore up the need for writeable or persistent
storage via PersistentVolumes
Read-only Root Filesystems

15. • Strong centralized
control over network
flow and encryption
• Accomplished via
sidecar containers in
every pod
Service Meshes
(Larger version of this on next slide)

16. Service Mesh Example: Istio

18. • Ask the API Server to:
• stage or modify containers
• allow us to MitM network traffic
• run commands in containers we don't own
• Ask the Kubelet to:
• run commands in containers we don't own
• display details of all workloads running in the
cluster
References: What Kinds of Attacks Exist

19. • Interact with the Cloud Provider
• Obtain node's credentials from the Metadata API
• Gain Kubernetes authentication tokens from
cloud storage buckets
• Modify or create compute instances
• Modify or duplicate storage
• Interact with any API that the node can
References: What Attack Types Do Cloud Providers Add