Cloud Native platforms such as Kubernetes help developers to easily get started deploying and running their applications at scale. But as this access to compute starts to become ubiquitous, how you secure and maintain compliance standards in these environments becomes extremely important. In this talk, we'll cover the basics of securing Cloud Native platforms such as Kubernetes. We will also cover open source tools - such as Clair, Anchore, and Sysdig Falco - that can be used to maintain a secure computing environment. Attendees will walk away with a good understanding of the challenges of securing a Cloud Native platform and practical advice on using open source tools as part of their security strategy.

1. @mfdii
Michael Ducy, Sysdig
Open Source Security
Tools for Kubernetes

2. @mfdii
Layers Container Security
Infra, Build, Runtime
Container Security Challenges
Open Source Tools For:
- Infra
- Build
- Runtime
Agenda

3. @mfdii
Layers of Container Security
Runtime
Build
Infrastructure

4. @mfdii
Infrastructure
Host Security
Networking
Cluster Security
Container Runtime

5. @mfdii
Build
Image/Software Provenance
- Signed Images/Layers
- Artifact Signing
Vulnerability Management
- Upstream OS
- Application Vulnerabilities

6. @mfdii
Runtime
Service/Container Admittance
Secure Secrets
Anomaly Detection
Forensics

7. @mfdii
Decisions Pushed to Edge
Ephemeral Nature of Containers
Attack Surface
Resource Isolation
Challenges of Container Security

8. @mfdii
Infrastructure Security
Network Storage
Host
Cluster
Container Runtime

9. @mfdii
Infrastructure Security
Cluster:
- RBAC, Security Policies, Affinity
Host/Container Runtime:
- Seccomp, SELinux, AppArmor, Resource Constraints
Network:
- Service Mesh, Network Policy, Network Filtering
Orchestrator:
- kube-hunter, kube-bench, kubesec.io

10. @mfdii
Security Policies
Security Policies define:
- Access to host resources:
- Filesystem, Host Network, Namespaces
- User/Group of Container
- Read Only Filesystem
- Linux capabilities available:
- http://man7.org/linux/man-pages/man7/capabilities.7.html
- Seccomp, AppArmor, or SELinux profiles

11. @mfdii
Build Security
Network Storage
Host
Cluster
App Code
App Runtime
Libraries
OS
App Code
App Runtime
Libraries
OS
App Code
App Runtime
Libraries
OS
Container Runtime

12. @mfdii
Container Security
Developers
and
Source
Code
Build and
Automated
CI/CD
Deploy and
Runtime
Secure
Design and
Architecture
Static Code
Analysis
Source Code
Dependency
Checks
Build Artifact
Scanning
Software
Package
Dependency
Checks
Configuration
Checks
Best
Practices
Checks
Network
Ingress and
Egress
Runtime
Anomaly
Detection
Runtime
Deployment
Monitoring
Many Other

13. @mfdii
Container Security
Developers
and
Source
Code
Build and
Automated
CI/CD
Deploy and
Runtime
Secure
Design and
Architecture
Static Code
Analysis
Source Code
Dependency
Checks
Build Artifact
Scanning
Software
Package
Dependency
Checks
Configuration
Checks
Best
Practices
Checks
Network
Ingress and
Egress
Runtime
Anomaly
Detection
Runtime
Deployment
Monitoring
Many Other
Container Image

14. @mfdii
Container Image Scanning
Tools and services that, at a high level, should:
• Take as input (minimally) a built container image
• Analyze/inspect the contents of the image itself
• Perform various types of security, best practice, and compliance checks
• Result in a report, notification, or control decision based on analysis and
checks, mapped to identifiable container image content
Various tools exist, today we present the OSS Anchore Engine
• Container native
• Runs as a service with a broad API
• Distributed system
• Powerful and customizable policy-based checks for security, best-
practice, and other process compliance

15. @mfdii
Anchore Policy Checks
Image checks
• OS Packages (RPM, DEB, APK)
• 3rd party packages (NPM, GEM, JAVA, PY)
• File names and contents
• Build Metadata (DockerFile)
Security checks
• Software Vulnerabilities (OS Packages, 3rd party packages)
• Secrets/Keys search
Anchore policies are flexible - customizable and
tunable by the user!

16. @mfdii
Container Image Policy Scan in
CI/CD

17. @mfdii
docker.io/anchore/anchore-engine:latest
Anchore Engine: Architecture
External API Kubernetes Webhook
Catalog Policy EngineSimpleQueue
Analyzer Worker
CI/CD Users (CLI/API)
Database
API Tier
State Tier
Analysis Tier

18. @mfdii
Install Anchore: docker-compose
mkdir anchore
mkdir anchore/config
mkdir anchore/db
cd anchore
curl https://raw.githubusercontent.com/anchore/anchore-engine/master/scripts/docker-compose/docker-
compose.yaml > docker-compose.yaml
curl https://raw.githubusercontent.com/anchore/anchore-engine/master/scripts/docker-compose/config.yaml
> config/config.yaml
docker-compose up -d
docker run anchore/engine-cli:latest anchore-cli --u admin --p foobar --url
http://172.18.0.1:8228/v1 system status
Service analyzer (dockerhostid-anchore-engine, http://anchore-engine:8084): up
Service simplequeue (dockerhostid-anchore-engine, http://anchore-engine:8083): up
Service apiext (dockerhostid-anchore-engine, http://anchore-engine:8228): up
Service kubernetes_webhook (dockerhostid-anchore-engine, http://anchore-engine:8338): up
Service catalog (dockerhostid-anchore-engine, http://anchore-engine:8082): up
Service policy_engine (dockerhostid-anchore-engine, http://anchore-engine:8087): up
Engine DB Version: 0.0.7
Engine Code Version: 0.2.4

19. @mfdii
Install Anchore: Helm
helm install --name anchore-stack stable/anchore-engine
kubectl get pods
NAME READY STATUS RESTARTS AGE
anchore-stack-anchore-engine-core-5bf44cb6cd-zxx2k 1/1 Running 0 38m
anchore-stack-anchore-engine-worker-5f865c7bf-r72vs 1/1 Running 0 38m
anchore-stack-postgresql-76c87599dc-bbnxn 1/1 Running 0 38m
ANCHORE_CLI_USER=admin
ANCHORE_CLI_PASS=$(kubectl get secret --namespace default anchore-stack-anchore-engine -o
jsonpath="{.data.adminPassword}" | base64 --decode; echo)
kubectl run -i --tty anchore-cli --restart=Always --image anchore/engine-cli --env
ANCHORE_CLI_USER=admin --env ANCHORE_CLI_PASS=${ANCHORE_CLI_PASS} --env
ANCHORE_CLI_URL=http://anchore-stack-anchore-engine.default.svc.cluster.local:8228/v1/ /
anchore-cli system status

20. @mfdii
Using Anchore: Jenkins CI/CD

21. @mfdii
Using Anchore: Jenkins CI/CD

22. @mfdii
Using Anchore: CLI(scripting)
anchore-cli image add docker.io/library/debian:latest
…
anchore-cli --json image get docker.io/library/debian:latest | jq '.[0]["analysis_status"]'
"analyzing"
anchore-cli --json image get docker.io/library/debian:latest | jq '.[0]["analysis_status"]'
"analyzed"
anchore-cli evaluate check docker.io/library/debian:latest
Image Digest: sha256:a0cd2c88c5cc65499e959ac33c8ebab45f24e6348b48d8c34fd2308fcb0cc138
Full Tag: docker.io/library/debian:latest
Status: fail
Last Eval: 2018-07-28T21:42:42Z
Policy ID: 2c53a13c-1765-11e8-82ef-23527761d060
anchore-cli image vuln docker.io/library/debian:latest all
anchore-cli image content docker.io/library/debian:latest os
anchore-cli image content docker.io/library/debian:latest npm
…

23. @mfdii
Using Anchore: Kubernetes
Admission Control
Kubernetes 1.9 and above supports VaildatingAdmissionWebhooks
• Kubernetes Admission Controllers
General Process
• User sends deployment request to Kubernetes API
• Kubernetes send admission control request to custom validator service
• Service contacts Anchore Engine API to perform policy evaluation on
each image specified in the request
• Service responds with accept/deny
Full detail: Policy-based Image Validation For Kubernetes
With Anchore Engine by Vic Iglesias

24. @mfdii
Image Scanning + Runtime:
Sysdig Falco and Anchore Engine
docker run --rm -e ANCHORE_CLI_USER=admin -e ANCHORE_CLI_PASS=foobar -e
ANCHORE_CLI_URL=http://192.168.1.3:8228/v1 sysdig/anchore-falco
- macro: anchore_stop_policy_evaluation_containers
condition: container.image.id in
("52057de6c8d0d0143dfc71fde55e58edaf3ccc5c2212221a614f45283c5ab063","65bf726222e13b0ceff0bb20bb6f
0e99cbf403a7a1f611fdd2aadd0c8919bbcf","8626492fecd368469e92258dfcafe055f636cb9cbc321a5865a98a0a6c
99b8dd","e86d9bb526efa0b0401189d8df6e3856d0320a3d20045c87b4e49c8a8bdb22c1”)
- rule: Run Anchore Containers with Stop Policy Evaluation
desc: Detect containers which does not receive a positive Policy Evaluation from Anchore
Engine.
condition: evt.type=execve and proc.vpid=1 and container and
anchore_stop_policy_evaluation_containers
output: A stop policy evaluation container from anchore has started (%container.info
image=%container.image)
priority: INFO tags: [container]

25. @mfdii
Image Scanning + Runtime:
Anchore Webhook Notifications
Anchore Catalog
Service
Image Update
Monitor
Policy Evaluation
Monitor
Vulnerability Scan
Monitor
…
Anchore
Webhook
Consumer
Email / Slack
Notify
New Build Trigger
Block/Undeploy
…
Anchore Webhook Notification

26. @mfdii
Runtime Security
Network Storage
Host
Cluster
App Code
App Runtime
Libraries
OS
App Code
App Runtime
Libraries
OS
App Code
App Runtime
Libraries
OS
Container Runtime

27. @mfdii
Runtime Security
Service/Container Admittance
- What’s Allowed to Run/Join a Service
Secure Secrets
- How do applications authenticate
Anomaly Detection
- Is my runtime environment being tampered with?
Forensics
- What happened if something was compromised?

28. @mfdii
Anomaly Detection
- Containers are isolated processes.
- Processes are “scoped” as to what’s expected.
- Container images are immutable, runtime environments
often aren’t.
- How do you detect “abnormal” behavior.

29. @mfdii
Falco: A CNCF Sandbox Project
Runtime Security for Cloud Native Platforms.
- Detect abnormal behavior in applications,
containers, and hosts.
- Audit system activity
Cloud Native Computing Foundation
Sandbox Level Project
- https://sysdig.com/blog/falco-cncf-sandbox/

30. @mfdii
Falco
A behavioral activity monitor
•Detects suspicious activity defined by a set of rules
•Uses Sysdig’s flexible and powerful filtering expressions
With full support for containers/orchestration
•Utilizes sysdig’s container & orchestrator support
And flexible notification methods
•Alert to files, standard output, syslog, programs
Open Source
•Anyone can contribute rules or improvements

31. Quick examples
A shell is run in a container container.id != host and proc.name = bash
Overwrite system binaries
fd.directory in (/bin, /sbin, /usr/bin, /usr/sbin)
and write
Container namespace change
evt.type = setns and not proc.name in
(docker, sysdig)
Non-device files written in /dev
(evt.type = create or evt.arg.flags contains O_CREAT)
and proc.name != blkid and fd.directory = /dev and
fd.name != /dev/null
Process tries to access camera
evt.type = open and fd.name = /dev/video0
and not proc.name in (skype, webex)

32. Falco architecture
falco_probe
Kernel
Module
Kernel
User
Syscalls
Sysdig Libraries
`
Events
Alerting
Falco Rules
Suspicious
Events File
Syslog
Stdout
Filter Expression
Shell

33. Falco Rules
25 common rules available OOTB
Focused on common container best practices:
■ Writing files in bin or etc directories
■ Reading sensitive files
■ Binaries being executed other than CMD/ENTRYPOINT

34. Falco rules
.yaml file containing Macros, Lists, and Rules
- macro: bin_dir
condition: fd.directory in (/bin, /sbin, /usr/bin, /usr/sbin)
- list: shell_binaries
items: [bash, csh, ksh, sh, tcsh, zsh, dash]
- rule: write_binary_dir
desc: an attempt to write to any file below a set of binary directories
condition: bin_dir and evt.dir = < and open_write and not package_mgmt_procs
output: "File below a known binary directory opened for writing
(user=%user.name command=%proc.cmdline file=%fd.name)"
priority: WARNING

35. @mfdii
How can you use Falco?

36. @mfdii
Response Engine &
Security Playbooks
● Detect abnormal events with Falco
● Publish alerts to Pub/Sub service (NATS.io)
● Subscribers can subscribe to various FALCO topics to receive alerts:
○ FALCO.* - All alerts
○ FALCO.Notice - Alerts of priority “Notice” only
○ FALCO.Critical - Alerts of priority “Critical” only
● Subscribers can take action on alerts:
○ Kill offending Pod
○ Taint Nodes to prevent scheduling
○ Isolate Pod with Networking Policy
○ Send notification via Slack

37. @mfdii
Response Engine &
Security Playbooks

38. @mfdii
Response Engine &
Security Playbooks
https://aws.amazon.com/blogs/opensource/securing-amazon-eks-lambda-falco/

39. @mfdii
Response Engine &
Security Playbooks
Detects abnormal event,
Publishes alert to NATS
Subscribers receive
Falco Alert through
NATS Server
Kubeless receives
Falco Alert, firing a
function to delete the
offending Kubernetes
Pod
https://sysdig.com/blog/oss-container-security-runtime/

40. @mfdii
Functions for Operations
- Easily write simple functions to react to security events
- Multiple subscribers can take multiple actions
- One function to delete a pod
- One function to notify teams
- One function to log events
- Small, reusable components

41. @mfdii
SIEM with EFK
● Security Information and Event Management
○ Collect security events
○ Easily allow reporting and correlation of events across various data sources
● Elasticsearch, Fluentd, Kibana
○ Fluentd - Cloud Native log aggregation
○ Elasticsearch - Schema free JSON data store
○ Kibana - powerful data visualization tool for Elasticsearch
● https://sysdig.com/blog/kubernetes-security-logging-fluentd-falco/

42. @mfdii
SIEM with EFK
Detects abnormal event,
Publishes alert to stdout
Fluentd ships alerts
to Elasticsearch
Kibana dashboards
can be used to
aggregate, filter, and
report on alerts.

43. @mfdii
SIEM with EFK

44. Join the community
• Website
•https://falco.org
•https://anchore.com/opensource
• Public Slack
•http://slack.sysdig.com/
•https://anchore.com/slack
•https://sysdig.slack.com/messages/falco
• Blog
•https://sysdig.com/blog/tag/falco/
•https://anchore.com/opensource

45. Learn more
Documentation
• Anchore Documentation
• Falco Documentation
Github
• https://github.com/falcosecurity/falco
• https://github.com/anchore/anchore-engine
Docker Hub
• https://hub.docker.com/r/sysdig/falco/
• https://hub.docker.com/r/anchore/anchore-engine/

46. @mfdii
Thank You.
Questions?
michael@sysdig.com
nurmi@anchore.com
bencer@sysdig.com