Secure your K8s cluster from multi-layers

1. Secure your K8s cluster
from multi-layers
Jiantang Hao

2. Agend
• About me
• K8s bug report
• Why kubernetes make security difficult
• Infrastructure Layer
• K8s Control Plane Layer
• K8s Workload Layer
• K8s Container Runtime Layer
• User Misconfiguration Layer
• Useful Tool&Documents
• References

3. About me
• Jiantang Hao
• Platform Engineer at Yahoo! JAPAN
• Focus on CaaS(Container as a Service)
• Like Challengeable/Interesting tech

4. K8s bug report
• https://k8s.devstats.cncf.io/d/39/issues-opened-closed-by-
sig?orgId=1&var-period=w&var-sig_name=All&var-
kind_name=bug

5. Details of kubernetes in CVE
• https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=kubernetes

6. Why kubernetes make security difficult
• Traffic is everywhere. Containers can be dynamically
deployed across hosts or even clouds.
• Increased attack surface. Each container and have a
different attack surface and vulnerabilities which can be
exploited.
• Old security tool. Old models and tools for security will not
be able to keep up in a constantly changing container
environment.

7. Infrastructure Layer
• Turn on audit log
• Never expose a port, which doesn't need exposure
• Host the cluster in a private subnet or VPC if possible
• Limit SSH to kubernetes nodes, use “kubectl” more
• Limit the access to kube-api

8. K8s Control Plane Layer
• Enable RBAC, at least make --anonymous-auth false.
• Enable TLS among component connection
• Encrypting Secret Data at Rest
• Turn on K8s audit logging
• Reserve Compute Resources for System Daemons
• Admission Controllers

9. Admission Controllers
• Enable by set flag in Kubernetes API server
• Admission controllers may be “validating”, “mutating”, or
both
• This admission controller calls any validating webhooks
which match the request. Matching webhooks are called in
parallel
• Caution about using mutating webhooks

10. Admission Controllers

11. ValidatingAdmissionWebhook

12. ValidatingAdmissionWebhook

13. K8s Workload Layer
• Run Containers as a Non-Root User
• Run a Cluster-wide Pod Security Policy
• Create and Define Cluster Network Policies
• Use namespace for isolation
• Controlling which nodes pods may access
• Controlling the capabilities by setting Resource Quota
• Security Context

14. Pod Security Policy

15. Deploy Pod Security Policy Via RBAC
• First, a Role or ClusterRole needs to grant access
to use the desired policies.

16. Lessons about using PSP
• If you only want to grant usage for pods being run in the
namespace, you have to create RoleBinding per namespace
• The above lead to a lot of individual rules for different use-
cases and it is difficult to make it maintainable in the long
term.
• RBAC Authorization is based on a whitelist, it is hard to set
Blacklist-based PSP
• Open Policy Agent could solve Blacklist-based problems

17. K8s Container Runtime Layer

18. Kata Containers

19. Kata Containers

20. Multi-tenant K8s Cluster

21. User Misconfiguration Layer
• One recent study found that 70‒75% of companies have at
least one serious cloud security misconfiguration
Image from https://compliancex.com/embarrassing-6bn-fat-finger-trade-another-blow-to-top-firm/

22. User Misconfiguration Layer
• Donʼt specify default values unnecessarily
• Simple, minimal configuration will make errors less likely
• Put object descriptions in annotations, to allow better
introspection
• Specify the latest stable API version
• Check the configuration on CICD pipeline
• https://kubesec.io/

23. Useful Tool&Documents
• (CIS) Benchmark for Kubernetes
• (CIS) Benchmark for Docker
• aquasecurity/kube-bench
• aquasecurity/kube-hunter
• Sysdig Inspect
• Shopify/kubeaudit
• coreos/clair
• Open Policy Agent

24. References• https://www.cncf.io/blog/2019/01/14/9-kubernetes-security-best-practices-everyone-must-follow/
• https://neuvector.com/container-security/kubernetes-security-guide/
• https://www.darkreading.com/vulnerabilities---threats/security-at-the-speed-of-devops-maturity-orchestration-
and-detection/a/d-id/1333583
• https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=kubernetes
• https://github.com/freach/kubernetes-security-best-practice
• https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#imagepolicywebhook
• https://kubernetes.io/docs/tasks/debug-application-cluster/audit/
• https://kubernetes.io/blog/2018/07/18/11-ways-not-to-get-hacked/s
• https://kubernetes-security.info/#securing-the-cluster
• https://medium.com/devopslinks/kubernetes-security-are-your-container-doors-open-2c4b99c8d786
• https://kubernetes.io/docs/concepts/configuration/overview/
• https://kubedex.com/kubernetes-container-runtimes/
• https://katacontainers.io/
• https://katacontainers.io/media/uploads/katacontainers/uploads/katacontainers/kata-containers-on-boarding-
deck-for-website01022018.pdf
• https://katacontainers.io/media/uploads/katacontainers/uploads/katacontainers/kata_containers_overview.pdf
• https://itnext.io/kubernetes-authorization-via-open-policy-agent-a9455d9d5ceb
• https://kubernetes.io/docs/concepts/policy/pod-security-policy/

25. Yes! We are hiring!
jhao@yahoo-corp.jp
https://about.yahoo.co.jp/hr/en/
https://about.yahoo.co.jp/hr/

26. Thank you for your listening！
ご静聴ありがとうございました！