SlideShare ist ein Scribd-Unternehmen logo
1 von 59
Downloaden Sie, um offline zu lesen
Container's
Anatomy
Namespaces, cgroups,
and some filesystem magic
1 / 59
Who am I?
Jérôme Petazzoni (@jpetazzo)
French software engineer living in California
I have built and scaled the dotCloud PaaS
(almost 5 years ago, time flies!)
I talk (too much) about containers
(first time was maybe in February 2013, at SCALE in L.A.)
Proud member of the House of Bash
(when annoyed, we replace things with tiny shell scripts)
2 / 59
Outline
What is a container?
(high level and low level overviews)
The building blocks
(namespaces, cgroups, copy-on-write storage)
Container runtimes
(Docker, LXC, rkt, runC, systemd-nspawn...)
Other containers
(OpenVZ, Jails, Zones...)
Hand-crafted, artisan-pick, house-blend containers
(with demos!)
3 / 59
What is a
container?
4 / 59
High level approach:
it's a lightweight VM
I can get a shell on it
(through SSH or otherwise)
It "feels" like a VM:
own process space
own network interface
can run stuff as root
can install packages
can run services
can mess up routing, iptables...
5 / 59
Low level approach:
it's chroot on steroids
It's not quite like a VM:
uses the host kernel
can't boot a different OS
can't have its own modules
doesn't need initas PID 1
doesn't need syslogd, cron...
It's just a bunch of processes visible on the host machine
(contrast with VMs which are opaque)
6 / 59
How are they implemented?
Let's look in the kernel source!
Go to LXR
Look for "LXC" → zero result
Look for "container" → 1000+ results
Almost all of them are about data structures
(or other unrelated concepts like "ACPI containers")
There are some references to "our" containers,
in documentation
7 / 59
How are they implemented?
Let's look in the kernel source!
Go to LXR
Look for "LXC" → zero result
Look for "container" → 1000+ results
Almost all of them are about data structures
(or other unrelated concepts like "ACPI containers")
There are some references to "our" containers,
in documentation
??? Are containers even in the kernel ???
8 / 59
Ancient archeology
Five years ago...
When using the LXC tools (lxc-start, lxc-stop...)
you would often get weird messages mentioning
/cgroup/container_name/...
The cgrouppseudo filesystem has counters for CPU, RAM,
I/O, and device access (/dev/*) but nothing about network
I didn't find about "namespaces" by myself
(maybe noticing /proc/$PID/nswould have helped;
more on that later)
9 / 59
Modern archeology
Nowadays:
Cgroups are often in /sys/fs/cgroup
You quickly stumble on nsenter
(which tips you off about namespaces)
There is significantly more documentation everywhere
10 / 59
The building
blocks
11 / 59
Control groups
12 / 59
Control groups
Resource metering and limiting
memory
CPU
block I/O
network*
Device node (/dev/*) access control
*With cooperation from iptables/tc; more on that later
13 / 59
Generalities
Each subsystem (memory, CPU...) has a hierarchy (tree)
Hierarchies are independent
(the trees for e.g. memory and CPU can be different)
Each process belongs to exactly 1 node in each hierarchy
(think of each hierarchy as a different dimension or axis)
Each hierarchy starts with 1 node (the root)
All processes initially belong to the root of each hierarchy*
Each node = group of processes
(sharing the same resources)
*It's a bit more subtle; more on that later
14 / 59
Example
cpu memory/
├── batch ├── 109
│  ├── bitcoins ├── 25
│  │  └── 52 ├── 26
│  └── hadoop ├── 27
│   ├── 109 ├── 52
│   └── 88 ├── 88
└── realtime └── databases
├── nginx ├── 1008
│  ├── 25 └── 524
│  ├── 26
│  └── 27
├── postgres
│  └── 524
└── redis
└── 1008
15 / 59
Memory cgroup: accounting
Keeps track of pages used by each group:
file (read/write/mmap from block devices)
anonymous (stack, heap, anonymous mmap)
active (recently accessed)
inactive (candidate for eviction)
Each page is "charged" to a group
Pages can be shared across multiple groups
(e.g. multiple processes reading from the same files)
When pages are shared, the groups "split the bill"
16 / 59
Memory cgroup: limits
Each group can have (optional) hard and soft limits
Soft limits are not enforced
(they influence reclaim under memory pressure)
Hard limits will trigger a per-group OOM killer
The OOM killer can be customized (oom-notifier);
when the hard limit is exceeded:
freeze all processes in the group
notify user space (instead of going rampage)
we can kill processes, raise limits, migrate containers ...
when we're in the clear again, unfreeze the group
Limits can be set for physical, kernel, total memory
17 / 59
Memory cgroup: tricky details
Each time the kernel gives a page to a process,
or takes it away, it updates the counters
This adds some overhead
Unfortunately, this cannot be enabled/disabled per process
(it has to be done at boot time)
Cost sharing means thata process leaving a group
(e.g. because it terminates) can theoretically cause
an out of memory condition
18 / 59
Cpu cgroup
Keeps track of user/system CPU time
Keeps track of usage per CPU
Allows to set weights
Can't set CPU limits
OK, let's say you give N%
then the CPU throttles to a lower clock speed
now what?
same if you give a time slot
instructions? their exec speed varies wildly
¯_ツ_/¯
19 / 59
Cpuset cgroup
Pin groups to specific CPU(s)
Reserve CPUs for specific apps
Avoid processes bouncing between CPUs
Also relevant for NUMA systems
Provides extra dials and knobs
(per zone memory pressure, process migration costs...)
20 / 59
Blkio cgroup
Keeps track of I/Os for each group
per block device
read vs write
sync vs async
Set throttle (limits) for each group
per block device
read vs write
ops vs bytes
Set relative weights for each group
Full disclosure: this used to be clunky for async operations, and I haven't tested it in ages. ☹
21 / 59
Net_cls and net_prio cgroup
Automatically set traffic class or priority,
for traffic generated by processes in the group
Only works for egress traffic
Net_cls will assign traffic to a class
(that has to be matched with tc/iptables,
otherwise traffic just flows normally)
Net_prio will assign traffic to a priority
(priorities are used by queuing disciplines)
22 / 59
Devices cgroup
Controls what the group can do on device nodes
Permissions include read/write/mknod
Typical use:
allow /dev/{tty,zero,random,null}...
deny everything else
A few interesting nodes:
/dev/net/tun(network interface manipulation)
/dev/fuse(filesystems in user space)
/dev/kvm(VMs in containers, yay inception!)
/dev/dri(GPU)
23 / 59
Subtleties
PID 1 is placed at the root of each hierarchy
When a process is created, it is placed in the same groups
as its parent
Groups are materialized by one (or multiple) pseudo-fs
(typically mounted in /sys/fs/cgroup)
Groups are created by mkdirin the pseudo-fs
To move a process:
echo$PID>/sys/fs/cgroup/.../tasks
The cgroup wars: systemd vs cgmanager vs ...
24 / 59
Namespaces
25 / 59
Namespaces
Provide processes with their own view of the system
Cgroups = limits how much you can use;
namespaces = limits what you can see (and therefore use)
Multiple namespaces:
pid
net
mnt
uts
ipc
user
Each process is in one namespace of each type
26 / 59
Pid namespace
Processes within a PID namespace only see processes in
the same PID namespace
Each PID namespace has its own numbering
(starting at 1)
When PID 1 goes away, the whole namespace is killed
Those namespaces can be nested
A process ends up having multiple PIDs
(one per namespace in which its nested)
27 / 59
Net namespace: in theory
Processes within a given network namespace
get their own private network stack, including:
network interfaces (including lo)
routing tables
iptables rules
sockets (ss, netstat)
You can move a network interface from a netns to another
iplinksetdeveth0netnsPID
28 / 59
Net namespace: in practice
Typical use-case:
use vethpairs
(two virtual interfaces acting as a cross-over cable)
eth0in container network namespace
paired with vethXXXin host network namespace
all the vethXXXare bridged together
(Docker calls the bridge docker0)
But also: the magic of --netcontainer
shared localhost(and more!)
29 / 59
Mnt namespace
Processes can have their own root fs (à la chroot)
Processes can also have "private" mounts
/tmp(scoped per user, per service...)
Masking of /proc, /sys
NFS auto-mounts (why not?)
Mounts can be totally private, or shared
No easy way to pass along a mount
from a namespace to another ☹
30 / 59
Uts namespace
gethostname / sethostname
'nuff said!
31 / 59
Ipc namespace
32 / 59
Ipc namespace
Does anybody knows about IPC?
33 / 59
Ipc namespace
Does anybody knows about IPC?
Does anybody cares about IPC?
34 / 59
Ipc namespace
Does anybody knows about IPC?
Does anybody cares about IPC?
Allows a process (or group of processes) to have own:
IPC semaphores
IPC message queues
IPC shared memory
... without risk of conflict with other instances.
35 / 59
User namespace
Allows to map UID/GID; e.g.:
UID 0→1999 in container C1 is mapped to
UID 10000→11999 on host
UID 0→1999 in container C2 is mapped to
UID 12000→13999 on host
etc.
Avoids extra configuration in containers
UID 0 (root) can be squashed to a non-privileged user
Security improvement
But: devil is in the details
36 / 59
Namespace manipulation
Namespaces are created with the clone()system call
(i.e. with extra flags when creating a new process)
Namespaces are materialized by pseudo-files in
/proc/<pid>/ns
When the last process of a namespace exits, it is destroyed
(but can be preserved by bind-mounting the pseudo-file)
It is possible to "enter" a namespace with setns()
(exposed by the nsenterwrapper in util-linux)
37 / 59
Copy-on-write
38 / 59
Copy-on-write storage
Create a new container instantly
(instead of copying its whole filesystem)
Storage keeps track of what has changed
Many options available
AUFS, overlay (file level)
device mapper thinp (block level)
BTRFS, ZFS (FS level)
Considerably reduces footprint and "boot" times
See also: Deep dive into Docker storage drivers
39 / 59
Other details
40 / 59
Orthogonality
All those things can be used independently
Use a few cgroups if you just need resource isolation
Simulate a network of routers with network namespaces
Put the debugger in a container's namespaces,
but not its cgroups (to not use its resource quotas)
Setup a network interface in an isolated environment,
then move it to another
etc.
41 / 59
One word about overhead
Even when you don't run containers ...
... you are in a container
Your host processes still execute in the root
namespaces and cgroups
Remember: there are three kind of lies
42 / 59
One word about overhead
Even when you don't run containers ...
... you are in a container
Your host processes still execute in the root
namespaces and cgroups
Remember: there are three kind of lies
Lies, damn lies, and benchmarks
43 / 59
Some missing bits
Capabilities
break down "root / non-root" into fine-grained rights
allow to keep root, but without the dangerous bits
however: CAP_SYS_ADMINremains a big catchall
SELinux / AppArmor ...
containers that actually contain
deserve a whole talk on their own
44 / 59
Container
runtimes
45 / 59
LXC
Set of userland tools
A container is a directory in /var/lib/lxc
Small config file + root filesystem
Early versions had no support for CoW
Early versions had no support to move images around
Requires significant amount of elbow grease
(easy for sysadmins/ops, hard for devs)
46 / 59
systemd-nspawn
From its manpage:
"For debugging, testing and building"
"Similar to chroot, but more powerful"
"Implements the Container Interface"
Seems to position itself as plumbing
47 / 59
systemd-nspawn
From its manpage:
"For debugging, testing and building"
"Similar to chroot, but more powerful"
"Implements the Container Interface"
Seems to position itself as plumbing
Recently added support for ɹǝʞɔop images
48 / 59
systemd-nspawn
From its manpage:
"For debugging, testing and building"
"Similar to chroot, but more powerful"
"Implements the Container Interface"
Seems to position itself as plumbing
Recently added support for ɹǝʞɔop images
#defineINDEX_HOST"index.do"/*theURLwegetthedatafrom*/"cker.io"
#defineHEADER_TOKEN"X-Do"/*theHTTPheaderfortheauthtoken*/"cker-Token:"
#defineHEADER_REGISTRY"X-Do"/*theHTTPheaderfortheregistry*/"cker-Endpoints:"
49 / 59
Docker Engine
Daemon controlled by REST(ish) API
First versions shelled out to LXC,
now uses its own libcontainerruntime
Manages containers, images, builds, and more
Some people think it does too many things
50 / 59
rkt, runC
Back to the basics!
Focus on container execution
(no API, no image management, no build, etc.)
They implement different specifications:
rkt implements appc (App Container)
runC implements OCP (Open Container Project),
leverages Docker's libcontainer
51 / 59
Which one is best?
They all use the same kernel features
Performance will be exactly the same
Look at:
features
design
ecosystem
52 / 59
Other containers
53 / 59
OpenVZ
Also Linux
Older, but battle-tested
(e.g. Travis CI gives you root in OpenVZ)
Tons of neat features too
ploop (efficient block device for containers)
checkpoint/restore, live migration
venet (~more efficient veth)
Still developed
54 / 59
Jails / Zones
FreeBSD / Solaris
Coarser granularity than namespaces and cgroups
Strong emphasis on security
Great for hosting providers
Not so much for developers
(where's the equivalent of dockerrun-tiubuntu?)
Note: Solaris branded zones can run Linux binaries
55 / 59
Build your own
56 / 59
FOR EDUCATIONAL
PURPOSES ONLY
57 / 59
58 / 59
Thanks!
Questions?
@jpetazzo
@docker
59 / 59

Más contenido relacionado

Was ist angesagt?

Ceph Block Devices: A Deep Dive
Ceph Block Devices:  A Deep DiveCeph Block Devices:  A Deep Dive
Ceph Block Devices: A Deep DiveRed_Hat_Storage
 
High-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uringHigh-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uringScyllaDB
 
[KubeCon NA 2020] containerd: Rootless Containers 2020
[KubeCon NA 2020] containerd: Rootless Containers 2020[KubeCon NA 2020] containerd: Rootless Containers 2020
[KubeCon NA 2020] containerd: Rootless Containers 2020Akihiro Suda
 
Linux container, namespaces & CGroup.
Linux container, namespaces & CGroup. Linux container, namespaces & CGroup.
Linux container, namespaces & CGroup. Neeraj Shrimali
 
Docker storage drivers by Jérôme Petazzoni
Docker storage drivers by Jérôme PetazzoniDocker storage drivers by Jérôme Petazzoni
Docker storage drivers by Jérôme PetazzoniDocker, Inc.
 
Implementation &amp; Comparison Of Rdma Over Ethernet
Implementation &amp; Comparison Of Rdma Over EthernetImplementation &amp; Comparison Of Rdma Over Ethernet
Implementation &amp; Comparison Of Rdma Over EthernetJames Wernicke
 
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptx
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptxGrafana Mimir and VictoriaMetrics_ Performance Tests.pptx
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptxRomanKhavronenko
 
Docker and kubernetes_introduction
Docker and kubernetes_introductionDocker and kubernetes_introduction
Docker and kubernetes_introductionJason Hu
 
Introduction to linux containers
Introduction to linux containersIntroduction to linux containers
Introduction to linux containersGoogle
 
Kernel Recipes 2019 - ftrace: Where modifying a running kernel all started
Kernel Recipes 2019 - ftrace: Where modifying a running kernel all startedKernel Recipes 2019 - ftrace: Where modifying a running kernel all started
Kernel Recipes 2019 - ftrace: Where modifying a running kernel all startedAnne Nicolas
 
Monitoring Microservices
Monitoring MicroservicesMonitoring Microservices
Monitoring MicroservicesWeaveworks
 
containerd summit - Deep Dive into containerd
containerd summit - Deep Dive into containerdcontainerd summit - Deep Dive into containerd
containerd summit - Deep Dive into containerdDocker, Inc.
 
Disaggregating Ceph using NVMeoF
Disaggregating Ceph using NVMeoFDisaggregating Ceph using NVMeoF
Disaggregating Ceph using NVMeoFShapeBlue
 
PostgreSQL WAL for DBAs
PostgreSQL WAL for DBAs PostgreSQL WAL for DBAs
PostgreSQL WAL for DBAs PGConf APAC
 
Cloud Native PostgreSQL
Cloud Native PostgreSQLCloud Native PostgreSQL
Cloud Native PostgreSQLEDB
 
Ceph Performance and Sizing Guide
Ceph Performance and Sizing GuideCeph Performance and Sizing Guide
Ceph Performance and Sizing GuideJose De La Rosa
 
Container Performance Analysis
Container Performance AnalysisContainer Performance Analysis
Container Performance AnalysisBrendan Gregg
 
Diving Through The Layers: Investigating runc, containerd, and the Docker eng...
Diving Through The Layers: Investigating runc, containerd, and the Docker eng...Diving Through The Layers: Investigating runc, containerd, and the Docker eng...
Diving Through The Layers: Investigating runc, containerd, and the Docker eng...Phil Estes
 
K8s in 3h - Kubernetes Fundamentals Training
K8s in 3h - Kubernetes Fundamentals TrainingK8s in 3h - Kubernetes Fundamentals Training
K8s in 3h - Kubernetes Fundamentals TrainingPiotr Perzyna
 
Linux Crash Dump Capture and Analysis
Linux Crash Dump Capture and AnalysisLinux Crash Dump Capture and Analysis
Linux Crash Dump Capture and AnalysisPaul V. Novarese
 

Was ist angesagt? (20)

Ceph Block Devices: A Deep Dive
Ceph Block Devices:  A Deep DiveCeph Block Devices:  A Deep Dive
Ceph Block Devices: A Deep Dive
 
High-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uringHigh-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uring
 
[KubeCon NA 2020] containerd: Rootless Containers 2020
[KubeCon NA 2020] containerd: Rootless Containers 2020[KubeCon NA 2020] containerd: Rootless Containers 2020
[KubeCon NA 2020] containerd: Rootless Containers 2020
 
Linux container, namespaces & CGroup.
Linux container, namespaces & CGroup. Linux container, namespaces & CGroup.
Linux container, namespaces & CGroup.
 
Docker storage drivers by Jérôme Petazzoni
Docker storage drivers by Jérôme PetazzoniDocker storage drivers by Jérôme Petazzoni
Docker storage drivers by Jérôme Petazzoni
 
Implementation &amp; Comparison Of Rdma Over Ethernet
Implementation &amp; Comparison Of Rdma Over EthernetImplementation &amp; Comparison Of Rdma Over Ethernet
Implementation &amp; Comparison Of Rdma Over Ethernet
 
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptx
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptxGrafana Mimir and VictoriaMetrics_ Performance Tests.pptx
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptx
 
Docker and kubernetes_introduction
Docker and kubernetes_introductionDocker and kubernetes_introduction
Docker and kubernetes_introduction
 
Introduction to linux containers
Introduction to linux containersIntroduction to linux containers
Introduction to linux containers
 
Kernel Recipes 2019 - ftrace: Where modifying a running kernel all started
Kernel Recipes 2019 - ftrace: Where modifying a running kernel all startedKernel Recipes 2019 - ftrace: Where modifying a running kernel all started
Kernel Recipes 2019 - ftrace: Where modifying a running kernel all started
 
Monitoring Microservices
Monitoring MicroservicesMonitoring Microservices
Monitoring Microservices
 
containerd summit - Deep Dive into containerd
containerd summit - Deep Dive into containerdcontainerd summit - Deep Dive into containerd
containerd summit - Deep Dive into containerd
 
Disaggregating Ceph using NVMeoF
Disaggregating Ceph using NVMeoFDisaggregating Ceph using NVMeoF
Disaggregating Ceph using NVMeoF
 
PostgreSQL WAL for DBAs
PostgreSQL WAL for DBAs PostgreSQL WAL for DBAs
PostgreSQL WAL for DBAs
 
Cloud Native PostgreSQL
Cloud Native PostgreSQLCloud Native PostgreSQL
Cloud Native PostgreSQL
 
Ceph Performance and Sizing Guide
Ceph Performance and Sizing GuideCeph Performance and Sizing Guide
Ceph Performance and Sizing Guide
 
Container Performance Analysis
Container Performance AnalysisContainer Performance Analysis
Container Performance Analysis
 
Diving Through The Layers: Investigating runc, containerd, and the Docker eng...
Diving Through The Layers: Investigating runc, containerd, and the Docker eng...Diving Through The Layers: Investigating runc, containerd, and the Docker eng...
Diving Through The Layers: Investigating runc, containerd, and the Docker eng...
 
K8s in 3h - Kubernetes Fundamentals Training
K8s in 3h - Kubernetes Fundamentals TrainingK8s in 3h - Kubernetes Fundamentals Training
K8s in 3h - Kubernetes Fundamentals Training
 
Linux Crash Dump Capture and Analysis
Linux Crash Dump Capture and AnalysisLinux Crash Dump Capture and Analysis
Linux Crash Dump Capture and Analysis
 

Ähnlich wie Anatomy of a Container: Namespaces, cgroups & Some Filesystem Magic - LinuxCon

Advanced Namespaces and cgroups
Advanced Namespaces and cgroupsAdvanced Namespaces and cgroups
Advanced Namespaces and cgroupsKernel TLV
 
LXC Containers and AUFs
LXC Containers and AUFsLXC Containers and AUFs
LXC Containers and AUFsDocker, Inc.
 
Lightweight Virtualization: LXC containers & AUFS
Lightweight Virtualization: LXC containers & AUFSLightweight Virtualization: LXC containers & AUFS
Lightweight Virtualization: LXC containers & AUFSJérôme Petazzoni
 
Linux 开源操作系统发展新趋势
Linux 开源操作系统发展新趋势Linux 开源操作系统发展新趋势
Linux 开源操作系统发展新趋势Anthony Wong
 
Containerization is more than the new Virtualization: enabling separation of ...
Containerization is more than the new Virtualization: enabling separation of ...Containerization is more than the new Virtualization: enabling separation of ...
Containerization is more than the new Virtualization: enabling separation of ...Jérôme Petazzoni
 
Docker 原理與實作
Docker 原理與實作Docker 原理與實作
Docker 原理與實作kao kuo-tung
 
Introduction to containers
Introduction to containersIntroduction to containers
Introduction to containersNitish Jadia
 
Realizing Linux Containers (LXC)
Realizing Linux Containers (LXC)Realizing Linux Containers (LXC)
Realizing Linux Containers (LXC)Boden Russell
 
Scale11x lxc talk
Scale11x lxc talkScale11x lxc talk
Scale11x lxc talkdotCloud
 
The building blocks of docker.
The building blocks of docker.The building blocks of docker.
The building blocks of docker.Chafik Belhaoues
 
Immutable infrastructure with Docker and containers (GlueCon 2015)
Immutable infrastructure with Docker and containers (GlueCon 2015)Immutable infrastructure with Docker and containers (GlueCon 2015)
Immutable infrastructure with Docker and containers (GlueCon 2015)Jérôme Petazzoni
 
"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...
"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo..."Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...
"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...Yandex
 
Containerization Is More than the New Virtualization
Containerization Is More than the New VirtualizationContainerization Is More than the New Virtualization
Containerization Is More than the New VirtualizationC4Media
 
LibOS as a regression test framework for Linux networking #netdev1.1
LibOS as a regression test framework for Linux networking #netdev1.1LibOS as a regression test framework for Linux networking #netdev1.1
LibOS as a regression test framework for Linux networking #netdev1.1Hajime Tazaki
 
Talk 160920 @ Cat System Workshop
Talk 160920 @ Cat System WorkshopTalk 160920 @ Cat System Workshop
Talk 160920 @ Cat System WorkshopQuey-Liang Kao
 
Introduction to Docker (as presented at December 2013 Global Hackathon)
Introduction to Docker (as presented at December 2013 Global Hackathon)Introduction to Docker (as presented at December 2013 Global Hackathon)
Introduction to Docker (as presented at December 2013 Global Hackathon)Jérôme Petazzoni
 
Java in containers
Java in containersJava in containers
Java in containersMartin Baez
 
Linux or unix interview questions
Linux or unix interview questionsLinux or unix interview questions
Linux or unix interview questionsTeja Bheemanapally
 
Docker and-containers-for-development-and-deployment-scale12x
Docker and-containers-for-development-and-deployment-scale12xDocker and-containers-for-development-and-deployment-scale12x
Docker and-containers-for-development-and-deployment-scale12xrkr10
 

Ähnlich wie Anatomy of a Container: Namespaces, cgroups & Some Filesystem Magic - LinuxCon (20)

Advanced Namespaces and cgroups
Advanced Namespaces and cgroupsAdvanced Namespaces and cgroups
Advanced Namespaces and cgroups
 
.ppt
.ppt.ppt
.ppt
 
LXC Containers and AUFs
LXC Containers and AUFsLXC Containers and AUFs
LXC Containers and AUFs
 
Lightweight Virtualization: LXC containers & AUFS
Lightweight Virtualization: LXC containers & AUFSLightweight Virtualization: LXC containers & AUFS
Lightweight Virtualization: LXC containers & AUFS
 
Linux 开源操作系统发展新趋势
Linux 开源操作系统发展新趋势Linux 开源操作系统发展新趋势
Linux 开源操作系统发展新趋势
 
Containerization is more than the new Virtualization: enabling separation of ...
Containerization is more than the new Virtualization: enabling separation of ...Containerization is more than the new Virtualization: enabling separation of ...
Containerization is more than the new Virtualization: enabling separation of ...
 
Docker 原理與實作
Docker 原理與實作Docker 原理與實作
Docker 原理與實作
 
Introduction to containers
Introduction to containersIntroduction to containers
Introduction to containers
 
Realizing Linux Containers (LXC)
Realizing Linux Containers (LXC)Realizing Linux Containers (LXC)
Realizing Linux Containers (LXC)
 
Scale11x lxc talk
Scale11x lxc talkScale11x lxc talk
Scale11x lxc talk
 
The building blocks of docker.
The building blocks of docker.The building blocks of docker.
The building blocks of docker.
 
Immutable infrastructure with Docker and containers (GlueCon 2015)
Immutable infrastructure with Docker and containers (GlueCon 2015)Immutable infrastructure with Docker and containers (GlueCon 2015)
Immutable infrastructure with Docker and containers (GlueCon 2015)
 
"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...
"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo..."Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...
"Lightweight Virtualization with Linux Containers and Docker". Jerome Petazzo...
 
Containerization Is More than the New Virtualization
Containerization Is More than the New VirtualizationContainerization Is More than the New Virtualization
Containerization Is More than the New Virtualization
 
LibOS as a regression test framework for Linux networking #netdev1.1
LibOS as a regression test framework for Linux networking #netdev1.1LibOS as a regression test framework for Linux networking #netdev1.1
LibOS as a regression test framework for Linux networking #netdev1.1
 
Talk 160920 @ Cat System Workshop
Talk 160920 @ Cat System WorkshopTalk 160920 @ Cat System Workshop
Talk 160920 @ Cat System Workshop
 
Introduction to Docker (as presented at December 2013 Global Hackathon)
Introduction to Docker (as presented at December 2013 Global Hackathon)Introduction to Docker (as presented at December 2013 Global Hackathon)
Introduction to Docker (as presented at December 2013 Global Hackathon)
 
Java in containers
Java in containersJava in containers
Java in containers
 
Linux or unix interview questions
Linux or unix interview questionsLinux or unix interview questions
Linux or unix interview questions
 
Docker and-containers-for-development-and-deployment-scale12x
Docker and-containers-for-development-and-deployment-scale12xDocker and-containers-for-development-and-deployment-scale12x
Docker and-containers-for-development-and-deployment-scale12x
 

Mehr von Jérôme Petazzoni

Use the Source or Join the Dark Side: differences between Docker Community an...
Use the Source or Join the Dark Side: differences between Docker Community an...Use the Source or Join the Dark Side: differences between Docker Community an...
Use the Source or Join the Dark Side: differences between Docker Community an...Jérôme Petazzoni
 
Orchestration for the rest of us
Orchestration for the rest of usOrchestration for the rest of us
Orchestration for the rest of usJérôme Petazzoni
 
Docker : quels enjeux pour le stockage et réseau ? Paris Open Source Summit ...
Docker : quels enjeux pour le stockage et réseau ? Paris Open Source Summit ...Docker : quels enjeux pour le stockage et réseau ? Paris Open Source Summit ...
Docker : quels enjeux pour le stockage et réseau ? Paris Open Source Summit ...Jérôme Petazzoni
 
Making DevOps Secure with Docker on Solaris (Oracle Open World, with Jesse Bu...
Making DevOps Secure with Docker on Solaris (Oracle Open World, with Jesse Bu...Making DevOps Secure with Docker on Solaris (Oracle Open World, with Jesse Bu...
Making DevOps Secure with Docker on Solaris (Oracle Open World, with Jesse Bu...Jérôme Petazzoni
 
Containers, docker, and security: state of the union (Bay Area Infracoders Me...
Containers, docker, and security: state of the union (Bay Area Infracoders Me...Containers, docker, and security: state of the union (Bay Area Infracoders Me...
Containers, docker, and security: state of the union (Bay Area Infracoders Me...Jérôme Petazzoni
 
From development environments to production deployments with Docker, Compose,...
From development environments to production deployments with Docker, Compose,...From development environments to production deployments with Docker, Compose,...
From development environments to production deployments with Docker, Compose,...Jérôme Petazzoni
 
How to contribute to large open source projects like Docker (LinuxCon 2015)
How to contribute to large open source projects like Docker (LinuxCon 2015)How to contribute to large open source projects like Docker (LinuxCon 2015)
How to contribute to large open source projects like Docker (LinuxCon 2015)Jérôme Petazzoni
 
Containers, Docker, and Security: State Of The Union (LinuxCon and ContainerC...
Containers, Docker, and Security: State Of The Union (LinuxCon and ContainerC...Containers, Docker, and Security: State Of The Union (LinuxCon and ContainerC...
Containers, Docker, and Security: State Of The Union (LinuxCon and ContainerC...Jérôme Petazzoni
 
Microservices. Microservices everywhere! (At OSCON 2015)
Microservices. Microservices everywhere! (At OSCON 2015)Microservices. Microservices everywhere! (At OSCON 2015)
Microservices. Microservices everywhere! (At OSCON 2015)Jérôme Petazzoni
 
Deploy microservices in containers with Docker and friends - KCDC2015
Deploy microservices in containers with Docker and friends - KCDC2015Deploy microservices in containers with Docker and friends - KCDC2015
Deploy microservices in containers with Docker and friends - KCDC2015Jérôme Petazzoni
 
Containers: from development to production at DevNation 2015
Containers: from development to production at DevNation 2015Containers: from development to production at DevNation 2015
Containers: from development to production at DevNation 2015Jérôme Petazzoni
 
The Docker ecosystem and the future of application deployment
The Docker ecosystem and the future of application deploymentThe Docker ecosystem and the future of application deployment
The Docker ecosystem and the future of application deploymentJérôme Petazzoni
 
Docker: automation for the rest of us
Docker: automation for the rest of usDocker: automation for the rest of us
Docker: automation for the rest of usJérôme Petazzoni
 
Docker Non Technical Presentation
Docker Non Technical PresentationDocker Non Technical Presentation
Docker Non Technical PresentationJérôme Petazzoni
 
Introduction to Docker, December 2014 "Tour de France" Bordeaux Special Edition
Introduction to Docker, December 2014 "Tour de France" Bordeaux Special EditionIntroduction to Docker, December 2014 "Tour de France" Bordeaux Special Edition
Introduction to Docker, December 2014 "Tour de France" Bordeaux Special EditionJérôme Petazzoni
 
Introduction to Docker, December 2014 "Tour de France" Edition
Introduction to Docker, December 2014 "Tour de France" EditionIntroduction to Docker, December 2014 "Tour de France" Edition
Introduction to Docker, December 2014 "Tour de France" EditionJérôme Petazzoni
 
Containers, Docker, and Microservices: the Terrific Trio
Containers, Docker, and Microservices: the Terrific TrioContainers, Docker, and Microservices: the Terrific Trio
Containers, Docker, and Microservices: the Terrific TrioJérôme Petazzoni
 
Pipework: Software-Defined Network for Containers and Docker
Pipework: Software-Defined Network for Containers and DockerPipework: Software-Defined Network for Containers and Docker
Pipework: Software-Defined Network for Containers and DockerJérôme Petazzoni
 
Docker Tips And Tricks at the Docker Beijing Meetup
Docker Tips And Tricks at the Docker Beijing MeetupDocker Tips And Tricks at the Docker Beijing Meetup
Docker Tips And Tricks at the Docker Beijing MeetupJérôme Petazzoni
 
Introduction to Docker at Glidewell Laboratories in Orange County
Introduction to Docker at Glidewell Laboratories in Orange CountyIntroduction to Docker at Glidewell Laboratories in Orange County
Introduction to Docker at Glidewell Laboratories in Orange CountyJérôme Petazzoni
 

Mehr von Jérôme Petazzoni (20)

Use the Source or Join the Dark Side: differences between Docker Community an...
Use the Source or Join the Dark Side: differences between Docker Community an...Use the Source or Join the Dark Side: differences between Docker Community an...
Use the Source or Join the Dark Side: differences between Docker Community an...
 
Orchestration for the rest of us
Orchestration for the rest of usOrchestration for the rest of us
Orchestration for the rest of us
 
Docker : quels enjeux pour le stockage et réseau ? Paris Open Source Summit ...
Docker : quels enjeux pour le stockage et réseau ? Paris Open Source Summit ...Docker : quels enjeux pour le stockage et réseau ? Paris Open Source Summit ...
Docker : quels enjeux pour le stockage et réseau ? Paris Open Source Summit ...
 
Making DevOps Secure with Docker on Solaris (Oracle Open World, with Jesse Bu...
Making DevOps Secure with Docker on Solaris (Oracle Open World, with Jesse Bu...Making DevOps Secure with Docker on Solaris (Oracle Open World, with Jesse Bu...
Making DevOps Secure with Docker on Solaris (Oracle Open World, with Jesse Bu...
 
Containers, docker, and security: state of the union (Bay Area Infracoders Me...
Containers, docker, and security: state of the union (Bay Area Infracoders Me...Containers, docker, and security: state of the union (Bay Area Infracoders Me...
Containers, docker, and security: state of the union (Bay Area Infracoders Me...
 
From development environments to production deployments with Docker, Compose,...
From development environments to production deployments with Docker, Compose,...From development environments to production deployments with Docker, Compose,...
From development environments to production deployments with Docker, Compose,...
 
How to contribute to large open source projects like Docker (LinuxCon 2015)
How to contribute to large open source projects like Docker (LinuxCon 2015)How to contribute to large open source projects like Docker (LinuxCon 2015)
How to contribute to large open source projects like Docker (LinuxCon 2015)
 
Containers, Docker, and Security: State Of The Union (LinuxCon and ContainerC...
Containers, Docker, and Security: State Of The Union (LinuxCon and ContainerC...Containers, Docker, and Security: State Of The Union (LinuxCon and ContainerC...
Containers, Docker, and Security: State Of The Union (LinuxCon and ContainerC...
 
Microservices. Microservices everywhere! (At OSCON 2015)
Microservices. Microservices everywhere! (At OSCON 2015)Microservices. Microservices everywhere! (At OSCON 2015)
Microservices. Microservices everywhere! (At OSCON 2015)
 
Deploy microservices in containers with Docker and friends - KCDC2015
Deploy microservices in containers with Docker and friends - KCDC2015Deploy microservices in containers with Docker and friends - KCDC2015
Deploy microservices in containers with Docker and friends - KCDC2015
 
Containers: from development to production at DevNation 2015
Containers: from development to production at DevNation 2015Containers: from development to production at DevNation 2015
Containers: from development to production at DevNation 2015
 
The Docker ecosystem and the future of application deployment
The Docker ecosystem and the future of application deploymentThe Docker ecosystem and the future of application deployment
The Docker ecosystem and the future of application deployment
 
Docker: automation for the rest of us
Docker: automation for the rest of usDocker: automation for the rest of us
Docker: automation for the rest of us
 
Docker Non Technical Presentation
Docker Non Technical PresentationDocker Non Technical Presentation
Docker Non Technical Presentation
 
Introduction to Docker, December 2014 "Tour de France" Bordeaux Special Edition
Introduction to Docker, December 2014 "Tour de France" Bordeaux Special EditionIntroduction to Docker, December 2014 "Tour de France" Bordeaux Special Edition
Introduction to Docker, December 2014 "Tour de France" Bordeaux Special Edition
 
Introduction to Docker, December 2014 "Tour de France" Edition
Introduction to Docker, December 2014 "Tour de France" EditionIntroduction to Docker, December 2014 "Tour de France" Edition
Introduction to Docker, December 2014 "Tour de France" Edition
 
Containers, Docker, and Microservices: the Terrific Trio
Containers, Docker, and Microservices: the Terrific TrioContainers, Docker, and Microservices: the Terrific Trio
Containers, Docker, and Microservices: the Terrific Trio
 
Pipework: Software-Defined Network for Containers and Docker
Pipework: Software-Defined Network for Containers and DockerPipework: Software-Defined Network for Containers and Docker
Pipework: Software-Defined Network for Containers and Docker
 
Docker Tips And Tricks at the Docker Beijing Meetup
Docker Tips And Tricks at the Docker Beijing MeetupDocker Tips And Tricks at the Docker Beijing Meetup
Docker Tips And Tricks at the Docker Beijing Meetup
 
Introduction to Docker at Glidewell Laboratories in Orange County
Introduction to Docker at Glidewell Laboratories in Orange CountyIntroduction to Docker at Glidewell Laboratories in Orange County
Introduction to Docker at Glidewell Laboratories in Orange County
 

Último

3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud DataEric D. Schabell
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptxHansamali Gamage
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveIES VE
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsDianaGray10
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4DianaGray10
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosErol GIRAUDY
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechProduct School
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTopCSSGallery
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3DianaGray10
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024Brian Pichman
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingMAGNIntelligence
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024Brian Pichman
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and businessFrancesco Corti
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxNeo4j
 
20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kitJamie (Taka) Wang
 
The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)IES VE
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInThousandEyes
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameKapil Thakar
 

Último (20)

3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data
 
.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx.NET 8 ChatBot with Azure OpenAI Services.pptx
.NET 8 ChatBot with Azure OpenAI Services.pptx
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
 
Automation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projectsAutomation Ops Series: Session 2 - Governance for UiPath projects
Automation Ops Series: Session 2 - Governance for UiPath projects
 
UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4UiPath Studio Web workshop series - Day 4
UiPath Studio Web workshop series - Day 4
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenarios
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development Companies
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3
 
Planetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile BrochurePlanetek Italia Srl - Corporate Profile Brochure
Planetek Italia Srl - Corporate Profile Brochure
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
 
IT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced ComputingIT Service Management (ITSM) Best Practices for Advanced Computing
IT Service Management (ITSM) Best Practices for Advanced Computing
 
AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024AI Workshops at Computers In Libraries 2024
AI Workshops at Computers In Libraries 2024
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and business
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
 
20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kit
 
The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First Frame
 

Anatomy of a Container: Namespaces, cgroups & Some Filesystem Magic - LinuxCon

  • 2. Who am I? Jérôme Petazzoni (@jpetazzo) French software engineer living in California I have built and scaled the dotCloud PaaS (almost 5 years ago, time flies!) I talk (too much) about containers (first time was maybe in February 2013, at SCALE in L.A.) Proud member of the House of Bash (when annoyed, we replace things with tiny shell scripts) 2 / 59
  • 3. Outline What is a container? (high level and low level overviews) The building blocks (namespaces, cgroups, copy-on-write storage) Container runtimes (Docker, LXC, rkt, runC, systemd-nspawn...) Other containers (OpenVZ, Jails, Zones...) Hand-crafted, artisan-pick, house-blend containers (with demos!) 3 / 59
  • 5. High level approach: it's a lightweight VM I can get a shell on it (through SSH or otherwise) It "feels" like a VM: own process space own network interface can run stuff as root can install packages can run services can mess up routing, iptables... 5 / 59
  • 6. Low level approach: it's chroot on steroids It's not quite like a VM: uses the host kernel can't boot a different OS can't have its own modules doesn't need initas PID 1 doesn't need syslogd, cron... It's just a bunch of processes visible on the host machine (contrast with VMs which are opaque) 6 / 59
  • 7. How are they implemented? Let's look in the kernel source! Go to LXR Look for "LXC" → zero result Look for "container" → 1000+ results Almost all of them are about data structures (or other unrelated concepts like "ACPI containers") There are some references to "our" containers, in documentation 7 / 59
  • 8. How are they implemented? Let's look in the kernel source! Go to LXR Look for "LXC" → zero result Look for "container" → 1000+ results Almost all of them are about data structures (or other unrelated concepts like "ACPI containers") There are some references to "our" containers, in documentation ??? Are containers even in the kernel ??? 8 / 59
  • 9. Ancient archeology Five years ago... When using the LXC tools (lxc-start, lxc-stop...) you would often get weird messages mentioning /cgroup/container_name/... The cgrouppseudo filesystem has counters for CPU, RAM, I/O, and device access (/dev/*) but nothing about network I didn't find about "namespaces" by myself (maybe noticing /proc/$PID/nswould have helped; more on that later) 9 / 59
  • 10. Modern archeology Nowadays: Cgroups are often in /sys/fs/cgroup You quickly stumble on nsenter (which tips you off about namespaces) There is significantly more documentation everywhere 10 / 59
  • 13. Control groups Resource metering and limiting memory CPU block I/O network* Device node (/dev/*) access control *With cooperation from iptables/tc; more on that later 13 / 59
  • 14. Generalities Each subsystem (memory, CPU...) has a hierarchy (tree) Hierarchies are independent (the trees for e.g. memory and CPU can be different) Each process belongs to exactly 1 node in each hierarchy (think of each hierarchy as a different dimension or axis) Each hierarchy starts with 1 node (the root) All processes initially belong to the root of each hierarchy* Each node = group of processes (sharing the same resources) *It's a bit more subtle; more on that later 14 / 59
  • 15. Example cpu memory/ ├── batch ├── 109 │  ├── bitcoins ├── 25 │  │  └── 52 ├── 26 │  └── hadoop ├── 27 │   ├── 109 ├── 52 │   └── 88 ├── 88 └── realtime └── databases ├── nginx ├── 1008 │  ├── 25 └── 524 │  ├── 26 │  └── 27 ├── postgres │  └── 524 └── redis └── 1008 15 / 59
  • 16. Memory cgroup: accounting Keeps track of pages used by each group: file (read/write/mmap from block devices) anonymous (stack, heap, anonymous mmap) active (recently accessed) inactive (candidate for eviction) Each page is "charged" to a group Pages can be shared across multiple groups (e.g. multiple processes reading from the same files) When pages are shared, the groups "split the bill" 16 / 59
  • 17. Memory cgroup: limits Each group can have (optional) hard and soft limits Soft limits are not enforced (they influence reclaim under memory pressure) Hard limits will trigger a per-group OOM killer The OOM killer can be customized (oom-notifier); when the hard limit is exceeded: freeze all processes in the group notify user space (instead of going rampage) we can kill processes, raise limits, migrate containers ... when we're in the clear again, unfreeze the group Limits can be set for physical, kernel, total memory 17 / 59
  • 18. Memory cgroup: tricky details Each time the kernel gives a page to a process, or takes it away, it updates the counters This adds some overhead Unfortunately, this cannot be enabled/disabled per process (it has to be done at boot time) Cost sharing means thata process leaving a group (e.g. because it terminates) can theoretically cause an out of memory condition 18 / 59
  • 19. Cpu cgroup Keeps track of user/system CPU time Keeps track of usage per CPU Allows to set weights Can't set CPU limits OK, let's say you give N% then the CPU throttles to a lower clock speed now what? same if you give a time slot instructions? their exec speed varies wildly ¯_ツ_/¯ 19 / 59
  • 20. Cpuset cgroup Pin groups to specific CPU(s) Reserve CPUs for specific apps Avoid processes bouncing between CPUs Also relevant for NUMA systems Provides extra dials and knobs (per zone memory pressure, process migration costs...) 20 / 59
  • 21. Blkio cgroup Keeps track of I/Os for each group per block device read vs write sync vs async Set throttle (limits) for each group per block device read vs write ops vs bytes Set relative weights for each group Full disclosure: this used to be clunky for async operations, and I haven't tested it in ages. ☹ 21 / 59
  • 22. Net_cls and net_prio cgroup Automatically set traffic class or priority, for traffic generated by processes in the group Only works for egress traffic Net_cls will assign traffic to a class (that has to be matched with tc/iptables, otherwise traffic just flows normally) Net_prio will assign traffic to a priority (priorities are used by queuing disciplines) 22 / 59
  • 23. Devices cgroup Controls what the group can do on device nodes Permissions include read/write/mknod Typical use: allow /dev/{tty,zero,random,null}... deny everything else A few interesting nodes: /dev/net/tun(network interface manipulation) /dev/fuse(filesystems in user space) /dev/kvm(VMs in containers, yay inception!) /dev/dri(GPU) 23 / 59
  • 24. Subtleties PID 1 is placed at the root of each hierarchy When a process is created, it is placed in the same groups as its parent Groups are materialized by one (or multiple) pseudo-fs (typically mounted in /sys/fs/cgroup) Groups are created by mkdirin the pseudo-fs To move a process: echo$PID>/sys/fs/cgroup/.../tasks The cgroup wars: systemd vs cgmanager vs ... 24 / 59
  • 26. Namespaces Provide processes with their own view of the system Cgroups = limits how much you can use; namespaces = limits what you can see (and therefore use) Multiple namespaces: pid net mnt uts ipc user Each process is in one namespace of each type 26 / 59
  • 27. Pid namespace Processes within a PID namespace only see processes in the same PID namespace Each PID namespace has its own numbering (starting at 1) When PID 1 goes away, the whole namespace is killed Those namespaces can be nested A process ends up having multiple PIDs (one per namespace in which its nested) 27 / 59
  • 28. Net namespace: in theory Processes within a given network namespace get their own private network stack, including: network interfaces (including lo) routing tables iptables rules sockets (ss, netstat) You can move a network interface from a netns to another iplinksetdeveth0netnsPID 28 / 59
  • 29. Net namespace: in practice Typical use-case: use vethpairs (two virtual interfaces acting as a cross-over cable) eth0in container network namespace paired with vethXXXin host network namespace all the vethXXXare bridged together (Docker calls the bridge docker0) But also: the magic of --netcontainer shared localhost(and more!) 29 / 59
  • 30. Mnt namespace Processes can have their own root fs (à la chroot) Processes can also have "private" mounts /tmp(scoped per user, per service...) Masking of /proc, /sys NFS auto-mounts (why not?) Mounts can be totally private, or shared No easy way to pass along a mount from a namespace to another ☹ 30 / 59
  • 31. Uts namespace gethostname / sethostname 'nuff said! 31 / 59
  • 33. Ipc namespace Does anybody knows about IPC? 33 / 59
  • 34. Ipc namespace Does anybody knows about IPC? Does anybody cares about IPC? 34 / 59
  • 35. Ipc namespace Does anybody knows about IPC? Does anybody cares about IPC? Allows a process (or group of processes) to have own: IPC semaphores IPC message queues IPC shared memory ... without risk of conflict with other instances. 35 / 59
  • 36. User namespace Allows to map UID/GID; e.g.: UID 0→1999 in container C1 is mapped to UID 10000→11999 on host UID 0→1999 in container C2 is mapped to UID 12000→13999 on host etc. Avoids extra configuration in containers UID 0 (root) can be squashed to a non-privileged user Security improvement But: devil is in the details 36 / 59
  • 37. Namespace manipulation Namespaces are created with the clone()system call (i.e. with extra flags when creating a new process) Namespaces are materialized by pseudo-files in /proc/<pid>/ns When the last process of a namespace exits, it is destroyed (but can be preserved by bind-mounting the pseudo-file) It is possible to "enter" a namespace with setns() (exposed by the nsenterwrapper in util-linux) 37 / 59
  • 39. Copy-on-write storage Create a new container instantly (instead of copying its whole filesystem) Storage keeps track of what has changed Many options available AUFS, overlay (file level) device mapper thinp (block level) BTRFS, ZFS (FS level) Considerably reduces footprint and "boot" times See also: Deep dive into Docker storage drivers 39 / 59
  • 41. Orthogonality All those things can be used independently Use a few cgroups if you just need resource isolation Simulate a network of routers with network namespaces Put the debugger in a container's namespaces, but not its cgroups (to not use its resource quotas) Setup a network interface in an isolated environment, then move it to another etc. 41 / 59
  • 42. One word about overhead Even when you don't run containers ... ... you are in a container Your host processes still execute in the root namespaces and cgroups Remember: there are three kind of lies 42 / 59
  • 43. One word about overhead Even when you don't run containers ... ... you are in a container Your host processes still execute in the root namespaces and cgroups Remember: there are three kind of lies Lies, damn lies, and benchmarks 43 / 59
  • 44. Some missing bits Capabilities break down "root / non-root" into fine-grained rights allow to keep root, but without the dangerous bits however: CAP_SYS_ADMINremains a big catchall SELinux / AppArmor ... containers that actually contain deserve a whole talk on their own 44 / 59
  • 46. LXC Set of userland tools A container is a directory in /var/lib/lxc Small config file + root filesystem Early versions had no support for CoW Early versions had no support to move images around Requires significant amount of elbow grease (easy for sysadmins/ops, hard for devs) 46 / 59
  • 47. systemd-nspawn From its manpage: "For debugging, testing and building" "Similar to chroot, but more powerful" "Implements the Container Interface" Seems to position itself as plumbing 47 / 59
  • 48. systemd-nspawn From its manpage: "For debugging, testing and building" "Similar to chroot, but more powerful" "Implements the Container Interface" Seems to position itself as plumbing Recently added support for ɹǝʞɔop images 48 / 59
  • 49. systemd-nspawn From its manpage: "For debugging, testing and building" "Similar to chroot, but more powerful" "Implements the Container Interface" Seems to position itself as plumbing Recently added support for ɹǝʞɔop images #defineINDEX_HOST"index.do"/*theURLwegetthedatafrom*/"cker.io" #defineHEADER_TOKEN"X-Do"/*theHTTPheaderfortheauthtoken*/"cker-Token:" #defineHEADER_REGISTRY"X-Do"/*theHTTPheaderfortheregistry*/"cker-Endpoints:" 49 / 59
  • 50. Docker Engine Daemon controlled by REST(ish) API First versions shelled out to LXC, now uses its own libcontainerruntime Manages containers, images, builds, and more Some people think it does too many things 50 / 59
  • 51. rkt, runC Back to the basics! Focus on container execution (no API, no image management, no build, etc.) They implement different specifications: rkt implements appc (App Container) runC implements OCP (Open Container Project), leverages Docker's libcontainer 51 / 59
  • 52. Which one is best? They all use the same kernel features Performance will be exactly the same Look at: features design ecosystem 52 / 59
  • 54. OpenVZ Also Linux Older, but battle-tested (e.g. Travis CI gives you root in OpenVZ) Tons of neat features too ploop (efficient block device for containers) checkpoint/restore, live migration venet (~more efficient veth) Still developed 54 / 59
  • 55. Jails / Zones FreeBSD / Solaris Coarser granularity than namespaces and cgroups Strong emphasis on security Great for hosting providers Not so much for developers (where's the equivalent of dockerrun-tiubuntu?) Note: Solaris branded zones can run Linux binaries 55 / 59