Application running in containers provide a myriad of choices to the end developer. But how do you provide the necessary services to monitor and secure these applications running in platforms such as Kubernetes. This presentation covers some common sense principles to monitor and secure your Kubernetes based applications.

1. Monitoring & Securing
Microservices in
Kubernetes
Michael Ducy - @mfdii - Sysdig

2. 5 Principles
1. Monitor containers and what’s inside them.
2. Alert on service performance, not container performance.
3. Monitor APIs.
4. Secure the container runtime environment.
5. Provide incident and forensic analysis for containers.

3. Let’s talk about Microservices

4. A quick refresher
• Let’s define microservices:
The term "Microservice Architecture" has sprung up over the last few years to describe
a particular way of designing software applications as suites of independently
deployable services. While there is no precise definition of this architectural style, there
are certain common characteristics around organization around business capability,
automated deployment, intelligence in the endpoints, and decentralized control of
languages and data.
http://www.martinfowler.com/articles/microservices.html

5. Stop, It’s not SOA
• Services tend to have smaller concerns than SOA Services
• Architectural Concepts incorporates Innovations:
– In Infrastructure
– In Automation
– In Continuous Delivery
– In Development
– In Monitoring
http://www.martinfowler.com/articles/microservices.html

6. What you think it looks like
http://microservices.io/i/Microservice_Architecture.png

7. What it really looks like.
https://www.appcentrica.com/wp-content/uploads/2016/11/Microservices-Architecture-1.png

8. But it’s probably more like
https://i.stack.imgur.com/y27IV.png

9. Let’s talk about Containers

10. What’s a Container?
Containers ARE NOT lightweight VMs.

11. What’s a Container?
A combination of:
● An image
● Cgroups
● Namespaces
● Linux Security Modules:
○ SELinux, App armor, seccomp

12. Containers vs. Zones vs. Jails vs. VMs
Thanks to Jessie Frazelle (@jessfraz) for this slide https://goo.gl/7fVKFa
Containers
Cgroups
Namespaces
LSMs
Zones
First class
concept
Jails
First class
concept
VMs
First class
concept
Read more about this here: https://blog.jessfraz.com/post/containers-zones-jails-vms/

13. What’s a Container
The Old Way The New Way
Heavy, Non Portable Light, Easily Portable

14. What’s a Container
The Old Way The New Way
Heavy, Non Portable Light, Easily Portable
Application
Artifact
Virtual Machine
Artifact
Container
Artifact

15. cgroups
Control Groups
Limits the quantity of a resource that can be consumed.
Memory, CPU, Net, etc

16. namespaces
namespaces
Limits what resources can be accessed
Cgroup CLONE_NEWCGROUP Cgroup root directory
IPC CLONE_NEWIPC System V IPC, POSIX message queues
Network CLONE_NEWNET Network devices, stacks, ports, etc.
Mount CLONE_NEWNS Mount points
PID CLONE_NEWPID Process IDs
User CLONE_NEWUSER User and group IDs
UTS CLONE_NEWUTS Hostname and NIS domain name

17. Linux Security Modules
SELinux
System wide execution policy
AppArmor
System wide execution policy, focused on processes
Seccomp
Per process system call isolation

18. Why is it magic?
Container
Image
Container
Runtime
cgroup
namespace
LSM

19. Containers + Microservices

20. Containers + Microservices
Containers
Limited scope
Easily deployable
Immutable
Versionable
Microservices
Limited Scope
CD used for deployment
Service Guarantees
Versionable API

21. Containers make
Microservices easier.

22. https://i.stack.imgur.com/y27IV.png

23. https://www.appcentrica.com/wp-content/uploads/2016/11/Microservices-Architecture-1.png

24. The Five Principles

25. 5 Principles
1. Monitor containers and what’s inside them.
2. Alert on service performance, not container performance.
3. Monitor APIs.
4. Secure the container runtime environment.
5. Provide incident and forensic analysis for containers.

26. 5 Principles
1. Monitor containers and what’s inside them.
2. Alert on service performance, not container performance.
3. Monitor APIs.
4. Secure the container runtime environment.
5. Provide incident and forensic analysis for containers.

27. A Paradigm Shift
Traditional management
focuses on apps running
on VMs and Hosts.
Containers shift the
management focus to
apps running as isolated
process.
vs.

28. Monitor Containers & What’s Inside
Monitoring needs to focus on:
● Isolated processes in the context of the container
○ Resource limits
● Isolated processes in the context of the orchestrator
○ Resource limits
○ Scaling limits

29. Monitor Containers & What’s Inside
Monitoring needs to focus on:
● Monitoring the application itself.
● Discovery of metrics for your language runtime,
application services, database, etc
● Discovery of metrics exposed by developers
○ Prometheus, JMX, etc

30. 5 Principles
1. Monitor containers and what’s inside them.
2. Alert on service performance, not container performance.
3. Monitor APIs.
4. Secure the container runtime environment.
5. Provide incident and forensic analysis for containers.

31. Average VM life:
23 Days
Average container life:
2.5 Days
Alert on service performance,
not container performance.

32. Alert on service performance,
not container performance.
● Containers don’t last long.
● Orchestration systems should monitor individual containers and take action.
○ Resource limits are important!
● Operators and Developers should worry about:
“Is the thing doing the thing it’s supposed
to be doing, and in a timely manner”

33. 5 Principles
1. Monitor containers and what’s inside them.
2. Alert on service performance, not container performance.
3. Monitor APIs.
4. Secure the container runtime environment.
5. Provide incident and forensic analysis for containers.

34. Monitor your APIs

35. Monitor your APIs
● APIs are the primary communication path for your Microservices
● Monitoring API health is important to:
○ Catch bottlenecks in certain methods/functions/endpoints
○ See frequently used methods/functions/endpoints
○ Trace behavior of your application through multiple systems

36. 5 Principles
1. Monitor containers and what’s inside them.
2. Alert on service performance, not container performance.
3. Monitor APIs.
4. Secure the container runtime environment.
5. Provide incident and forensic analysis for containers.

37. Secure the Runtime Environment
Current container security paradigms:
● Image Based
○ Image Scanning
○ Trusted Images
● Orchestrator Security
○ RBAC
● Service Based
○ SPIFFE, Cilium, Service Mesh
● Runtime Security
○ Sysdig Secure, Sysdig Falco

38. Image Based Security
Image Scanning:
● Scan image for known vulnerabilities
● Still requires process for bringing in upstream security releases
● “Point in Time Compliance”
● Integrate into container build process
● Clair & Anchore
Trusted Images/Registries:
● Cryptographically Sign Images
○ Still “Point in Time”
● Only trust particular container registries.
○ Only as good as your container build process

39. Orchestrator Security
Kubernetes RBAC:
● Stable as of Kubernetes 1.8
● Rich ability to limit users access to Kubernetes resources
Securing APIs, Dashboards, etc:
● Common exploitation point
● Open API ports to internet
● Exposed dashboards

40. Service Based
Service Mesh
● Ingress, Egress, Service Discovery, Failure Handling, Service Identity
Admittance Control
● Service Mesh, SPIFFE
Kernel Level Access Control
● API Level Access Control
● Cilium

41. Runtime Security
● What happens when a container starts/runs?
● Container Images are Immutable
● Runtime environments often aren’t

42. Runtime Security
● Watches Containers for “Abnormal Behavior”
○ Rules defined by policy
■ CIS, PCI, Organizational Best Practice, etc
○ Rules defined by container baselining
● Action taken based on severity
○ Stop/Kill container
● Should be part of your CD pipeline
○ Ensure developers haven’t added anything new that voilates
runtime policy.

43. 5 Principles
1. Monitor containers and what’s inside them.
2. Alert on service performance, not container performance.
3. Monitor APIs.
4. Secure the container runtime environment.
5. Provide incident and forensic analysis for containers.

44. Incident & Forensics Analysis
Incident & Forensics traditionally difficult:
● Bare Metal - Save the Drives
● VM - Snapshot VM state, Keep VM
Containers make Incident & Forensics easier:
● Smaller footprint
● Limited objects in the container
Orchestrators make Incident & Forensics difficult:
● Ephemeral nature of containers
● Rapid deploy cycles

45. Incident & Forensics Analysis
Container Incident & Forensics Analysis requires:
● Capturing the container before it’s gone
● Capturing the system activity for analysis
● Correlation between abnormal behavior and system activity

46. To recap.
Containers change how we monitor applications.
Microservices change how we monitor services.
1. Monitor containers and what’s inside them.
2. Alert on service performance, not container performance.
3. Monitor APIs.
4. Secure the container runtime environment.
5. Provide incident and forensic analysis for containers.

47. Thank You!
Twitter:
- @sysdig - @mfdii
Sysdig Blog:
- https://sysdig.com/blog/
Sysdig Container Intelligence Platform:
- https://sysdig.com/
Sysdig Trial:
- https://sysdig.com/sign-up

48. Thank You