Dockerized containers are the current wave that promising to revolutionize IT. Everybody is talking about containers, but a lot of people remain confused on how they work and why they are different or better than virtual machines. In this session, Black Duck container and virtualization expert Tim Mackey will demystify containers, explain their core concepts, and compare and contrast them with the virtual machine architectures that have been the staple of IT for the last decade.
2. • An overview of why containers are so hot today
• How containers work
• Pitfalls in container development
Session Details
3. Quick review: Virtual machines
Operating
System
User Space
Ring 2Ring 2
Ring 1Ring 1
Kernel
User Space
Ring 2Ring 2
Ring 1Ring 1
Kernel
Hypervisor
User Space
Ring 2Ring 2
Ring 1Ring 1
Kernel
Real mode Protected mode Virtualization extensions
4. Container – Something which uses “Docker” to encapsulate a service
• Need not be Docker. Alternatives are CoreOS rkt and Open Container Initiative
Microservice – A service designed to perform a single task
• Think smallest unit of usable work to scale
Cloud Native Design – A paradigm where enterprise resiliency principles do not
apply (more on that later)
Pet – A term referring to a long lived service in an enterprise
• Pets are cared for, and typically loved, nurtured and evolve
Cattle – A term referring to a service whose welfare is not a priority
• The concept being that cattle are replaceable – sorry animal lovers
Container Vernacular
5. 1. Containers are immutable
• Create once, run many instances
1. Containers are ephemeral
• A containers’ life should be only as long as absolutely required
1. Containers can be sacrificed
• Orchestration systems may terminate a container if needed
• No guarantee of lifespan
• Don’t store data or logs in containers
1. Containers limit resource access
• Define cgroups for CPU/RAM access
• Avoid use of ROOT credentials
• Layered file system helps manage storage
Container Rules – Cloud Native Designs
6. Docker 1.10 and and later
What is a Container?
Linux Containers
Linux kernel
namespaces cgroupscgroups
SELinux/AppArmorSELinux/AppArmor
liblxc
Linux kernel
namespaces cgroupscgroups
SELinux/AppArmorSELinux/AppArmor
Docker EngineDocker Engine
runC
containerdcontainerd
runC runC
containerd-shim containerd-shim containerd-shim
7. Application containers
• Hold a single application
• Can follow micro-services design pattern
• Starting point for most container usage
• Short lifespan, many per host
System containers
• Proxy for a VM
• Insulate against core operating system
• Perfect for legacy apps
• Long lifespan, few per host
Container Use Cases
KernelKernel
KernelKernel
10. • Representation of an application
• Built from a “Dockerfile”
• Uses a layered file system
• Single process per application
• Created in a Docker engine
• Appears as local binary repository
• Stored in a binary repository known as a Registry
Container Image
11. • Where container images are stored outside of local engine
• Core commands
• docker tag puts a mutable tag on the image
• docker push uploads an image to the registry
• docker pull downloads an image from the registry
• docker.io is default (aka Docker Hub is “predefined” as trusted
• Be explicit about image source, and validate
Container Registry
16. How Images are Created – Image Layer Cache Impacts
Build
17. • Where does your base image actually come from?
• What is the health of that base image?
• You’re updating it at build time, but from what cache?
• You trust your build servers, but who controls them?
• Who has rights to modify container images?
• What happens if base image registry goes away?
• What happens if base image tag goes away?
• What happens if an update mirror goes down?
• How do I patch container images?
Container Trust Issues
18. Image Health is Critical to Image Security
Docker Hub Container Scanning Red Hat Container Catalog Health Index
19. Objective: Share data between container and outside world
Core commands
• docker volume create create the volume
• docker run myimage –v source:/data map source to /data within container
Supported drivers
• local, tmpfs, btrfs, nfs represent mounted filesystems on host
Use in Dockerfile
Docker Volume Management
Modifies build image - Wrong
FROM centos:centos7
VOLUME /data
RUN touch /data/x
Modifies data in built image - Correct
FROM centos:centos7
RUN mkdir /data && touch /data/x
VOLUME /data
20. Secrets are…
• Application encryption keys
• Access tokens
• Credentials
Secrets are managed by…
• Environment variables – small installations and developer scenarios
• JSON Web Tokens (JWT)
• Docker Compose in Swarm mode (Compose V3)
• External secret services (HashiCorp Vault, Keywhiz, Consul, Amazon KMS, etc)
Secret Management in Containers
22. docker run myimage -ti
• Runs a new interactive instance of container image `myimage` with default entry point
docker run myimage –ti /usr/bin/sh
• Runs a new interactive instance of container image `myimage` opening a shell
docker attach name
• Attaches to running instance of container named `name`
docker exec name –ti /usr/bin/sh
• Attaches to running instance of container named `name` and opens a shell
docker logs myimage --follow
• Attaches to running instance of container named `name` opening and following stdout
Container Debugging
24. Deployment Requirements – Trust Your Container Source
Minimal HostMinimal Host
RedHat Container
Catalog
Docker Hub Third Party and Custom
Problem: Who to trust, and why?
• Trusted source?
• Unexpected image contents
• Locked application layer
versions (e.g. no yum update)
• Layer dependencies
(monolithic vs micro-services)
• Validated when?
• Trusted platform
• Signed by whom?
25. • Docker default network is Linux Bridge
• Access policy defined in iptables
• Based on Docker daemon startup
• External communication on by default
• -- iptables=off to disable iptables modification
• Inter container communication on by default
• -- icc=false to disable inter container communication
• -- link=CONTAINER_NAME_or_ID:ALIAS with EXPOSE ports from Docker file
• All inter-container/cross host communication is external
• `docker network` command simplifies aspects of network design
• Create user defined networks, including overlay networks
• docker network create --driver bridge sql
Deployment – Define Sensible Network Policies
27. • Enable Linux Security Modules
• SELinux
• --selinux-enabled on Docker engine, --security-opt=“label:profile”
• AppArmor
• -- security-opt=“apparmor:profile”
• Apply Linux kernel security profiles
• grsecurity, PaX and seccomp protections for ALSR and RBAC
• Adjust privileged kernel capabilities
• Reduce capabilities with --cap-drop
• Beware –cap-add and –privileged=false, and CAP_SYS_ADMIN
• Use a minimal Linux Host OS
• Red Hat Linux Atomic Host, Container Linux, RancherOS
• Reduce impact of noisy neighbors
• Use cgroups to set CPU shares and memory
Secure the Host and Namespaces
If you assume from the outset your containers will be compromised, what would you do differently? What could you do to make life much harder to mount an attack from a compromised container?
The first item everyone should have on their list is to enable the SELinux and AppArmor security modules for the distro you’re using as a base. For SELinux, you specify that it’s enabled on the command line for Docker Engine, and then on the container you specify a security option representing the profile you want to use.
Now that you have some security modules in place, you’re in better shape, but you can still make it much harder to mount a proper attack. Consider for example an attacker who recognizes they’re in a container, and assumes that means there probably are multiple containers with the same profile. Armed with that knowledge, one of the easiest ways to make the life of an attacker harder is to randomize the memory load location. That’s where kernel security profiles come into play. Grsecurity, PaX and seccomp enable roles based access and address randomization upon load capabilities. Net of this, where the executable code lives in memory changes, and that makes it harder to know if you’ve created a viable attack or not.
Now the Docker people have done a great job over the years in locking down the level of privileged access you get from within a container. In part, this is done using what’s known as kernel capabilities. Kernel capabilities offer a granular level of control over the types of operations you want the kernel to perform. Consistent with the concept of least privilege, you don’t want to ask for more rights than you need. If you find the defaults are providing more access than your application requires, you can pare things back using the –cap-drop option. Of course, it’s entirely possible you might need more rights, but if you find you need to disable priviledged access, or want to set the CAP_SYS_ADMIN flag, beware you’re effectively giving the container the equivalent of root access.
Lastly, from a security perspective, you can choose to use a minimal Linux distro such as Atomic or CoreOS, but you still will want to pay attention to the options I’ve just outlined.
So now that you’ve limited the access a potential attacker has to system services, you still have to contend with other types of havoc. A perfect example of this is the concept of a noisy neighbor. Most of us have had the experience of having someone in a neighboring space behave in an annoying manner. In the case of computing, the annoying neighbor can be one which consumes excessive memory or processing time. Limiting the scope of interference is very easy. All you need to do is define some CPU shares and memory limits for the container and set them during launch.