1. Security Level:
Cgroups:
Understanding better!
Ashish Pandey, Liuyang, Roopesh Keeppattu, Sanil Kumar D.
www.huawei.com
OSI Days, November, 2011, Bangalore
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential

2. Agenda
•Setting the context…
- Background and Cgroups Definition
- Significance of Cgroups
• Cgroups: Deeper View
- Features and Design
- Cgroups Subsystems
- Cgroups Implementation
•Cgroups Use cases
- Usage Illustration
- Sample Use cases
•Cgroups : Challenges and Future
- Key Challenges
- Future
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011 2

3. Background
 Clustering and Aggregation of Resources help in Fairness in
Large Systems
 Predictability in Performance depends on the predictability in
the availability of resource
Isolation and Aggregation of Resources for specific activities can improve
the predictability of the performance for the specific activities. This also
helps in Resource Tracking, Control and Management
 Examples : Cloud, Virtualization, Containers
Clou
d
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011 3

4. Cgroups Definition
Cgroups helps for resource isolation and resource control
Cgroups: Process Cgroups are like process container or
Container
process groups in Linux kernel
Provide a mechanism for
aggregating / partitioning sets of
tasks, and all their future children, into
hierarchical groups with specialized
behaviour
P
P
Associate a set of tasks with a set of
P P
P P P P
P parameters for one or more
subsystems
Cgroups help tuning the system to make best use of available hardware and network resources.
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011 4

5. Significance of Cgroups
 As the systems are becoming more
complex with dynamic resource needs, better
resource management is inevitable to get
optimized outout.
 Resource planning is better than dynamic
resource allocation
 Cgroups with its tree-structured hierarchy
and subsystems achieves resource planning for
the native system
Mainly 2 types of Subsystems supported:
•Isolation and Special Controllers :
cpuset, namespace, freezer, device, checkpoint/restart
•Resource Controllers : cpu(scheduler), memory, disk I/O, network
The significance of Cgroups is that it helps to implement the essential resource groups
hierarchically in kernel to enable better resource management. Every task running in the system
is in exactly one of the Cgroups in this hierarchy
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011 5

6. Agenda
•Setting the context…
- Background and Cgroups Definition
- Significance of Cgroups
• Cgroups: Deeper View
- Features and Design
- Cgroups Subsystems
- Cgroups Implementation
•Cgroups Use cases
- Usage Illustration
- Sample Use cases
•Cgroups : Challenges and Future
- Key Challenges
- Future
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011 6

7. Features and Design
Resource Isolation Resource Control Resource Tracking
• Isolate the resources for • Controls the resource for • Resource Tracking to
specific similar task groups usage limits, freezing measure how much
• Separate groups or check pointing resources certain systems
Cpuset, Namespace, Device for and restarting use for e.g. billing purposes
groups • Freezer, Checkpoint / • State of the resource is
Restart, Cpuacct, CPU, Me tracked
mory, Disk I/O and
Network I/O
Hierarchical based grouping of similar tasks in terms of resources is the design principle behind
Cgroups
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011 7

8. Cgroups Subsystems
 Modules which makes use of task grouping facilities provided
by Cgroups to treat groups of tasks in particular way
 Resource controllers allocating different levels of system
resources to different Cgroups.
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011 8

9. Cgroups Subsystems Details
Subsystem Type Description
cpuset Isolation Use cpuset to tie processes to system subsets of CPU
For showing private view of system to processes in cgroup;
ns Isolation
mainly used for OS level virtualization
cpu Control Shares CPU bandwidth between groups
cpuacct Accounting Groups tasks using cgroup and accounts the CPU usage
memory Control Limits memory and memory related parameters
devices Isolation Decides the access to devices by tasks in cgroup
freezer Control Freezes/unfreezes tasks in a group. Used for HPC clusters
Identification of packets originating from a specific cgroup and
net_cls Control
actions on the packet.
controls and monitors access to I/O on block devices by tasks in
blk_io Control
cgroup
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011 9

10. Cgroups Implementation
 VFS based filesystem model implementation
 Each task in the system has a reference-counted
pointer to a css_set
 A css_set contains a set of reference-counted
pointers to cgroup_subsys_state objects, one for each
cgroup subsystem registered in the system
 A cgroup hierarchy filesystem can be mounted for
browsing and manipulation from user space
 You can list all the tasks (by pid) attached to any
cgroup [cat /cgroup/tasks OR cat /cgroup/<group>/tasks]
 Files:
• kernel/cgroup.c
• include/linux/cgroup.h
• include/linux/cgroup_subsys.h
• +various subsystems implementations
 cgroups requires a few, simple hooks into the rest of
the kernel, these hooks are not in performance critical
paths of the kernel.
• In init/main.c : To initialize C groups at boot
time.
• In fork and exit calls : To attach and detach a
task from cgroups
A new file system, of type "cgroup" should be mounted, to enable browsing and modifying the
all the registered Cgroups subsystem
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011 10

11. Cgroups : How to write a new subsystem
APIs to be implemented by a new subsystem
Major modifications to be done:
•init/Kconfig: kernel configuration
parameters (general setup)
•include/linux/cgroup_subsys.h:
cgroup subsystem definition
•kernel/cgroup_example.c:
cgroup subsystem implementation
[Implement Kernel APIs as per
cgroup_subsys]
•kernel/Makefile: Makefile of the
core kernel components
•Add the appropriate hooks into the
kernel
As Cgroups provide clean kernel APIs , new subsystem addition and management are simpler
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011 11

12. Agenda
•Setting the context…
- Background and Cgroups Definition
- Significance of Cgroups
• Cgroups: Deeper View
- Features and Design
- Cgroups Subsystems
- Cgroups Implementation
•Cgroups Use cases
- Usage Illustration
- Sample Use cases
•Cgroups : Challenges and Future
- Key Challenges
- Future
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011 12

13. Usage Illustration
mount -t tmpfs none /sys/fs/cgroup
mount -t cgroup -ocpuset cpuset mkdir /sys/fs/cgroup/memory
/sys/fs/cgroup/cpuset mount -t cgroup none /sys/fs/cgroup/memory -o
cd /sys/fs/cgroup/cpuset memory
mkdir Charlie #Make the new group and move bash into it
cd Charlie mkdir /sys/fs/cgroup/memory/0
/bin/echo 2-3 > cpuset.cpus echo $$ > /sys/fs/cgroup/memory/0/tasks
/bin/echo 1 > cpuset.mems #Since now we're in the 0 cgroup, we can alter the
/bin/echo $$ > tasks memory limit:
sh echo 4M >
# The subshell 'sh' is now running in /sys/fs/cgroup/memory/0/memory.limit_in_bytes
cpuset Charlie cat /sys/fs/cgroup/memory/0/memory.limit_in_bytes
# The next line should display 4194304
'/Charlie' #We can check the usage:
cat /proc/self/cpuset cat /sys/fs/cgroup/memory/0/memory.usage_in_bytes
1216512
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011 13

14. Usage Illustration
#An entry is added using
mkdir /sys/fs/cgroup/freezer
devices.allow, and removed using
mount -t cgroup -ofreezer freezer
devices.deny.
/sys/fs/cgroup/freezer
For #instance
mkdir /sys/fs/cgroup/freezer/0
echo 'c 1:3 mr' >
echo $some_pid > /sys/fs/cgroup/freezer/0/tasks
/sys/fs/cgroup/1/devices.allow
#to get status of the freezer subsystem :
#allows cgroup 1 to read and mknod
cat /sys/fs/cgroup/freezer/0/freezer.state
the device usually known as dev/null.
THAWED
#Doing
#to freeze all tasks in the container :
echo a >
echo FROZEN > /sys/fs/cgroup/freezer/0/freezer.state
/sys/fs/cgroup/1/devices.deny
#at /sys/fs/cgroup/freezer/0/freezer.state
#will remove the default 'a *:* rwm'
FROZEN
entry. Doing
#to unfreeze all tasks in the container :
echo a >
echo THAWED > /sys/fs/cgroup/freezer/0/freezer.state
/sys/fs/cgroup/1/devices.allow
cat /sys/fs/cgroup/freezer/0/freezer.state
#will add the 'a *:* rwm' entry to the
THAWED
whitelist.
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011 14

15. Sample Use cases
Without CPU cgroup subsystem With CPU cgroup subsystem
(10 tasks in “multimedia” and 5 tasks in “browser”) (10 tasks in “multimedia” and 5 tasks in “browser”)
Cgroups can provide better resource control and management
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011 15

16. Sample Use cases (contd.)
 A large university server with various users - students, professors, system tasks etc. The resource planning for
this server could be along the following lines with cgroups
Users
Professor Student
CPU (0-3) Disk (50%) Memory(50%) N/W CPU((4-7) Disk(50%) Memory(50%) N/W
WWW WWW
NFS(25%) Others(10%) NFS(25%) Others(10%)
browsing(20%) browsing(10%)
 Consider a batch job management system which requires start and stop sets of tasks in order to schedule the
resources of a machine .This kind of program is often used on HPC clusters to schedule access to the cluster as a
whole Batch
Processing
System
Batch Batch
Process1 Process2
Can be
Cpu(0-4) Disk(50%) Memory(50%) No Freeze Cpu(5-8) Disk(40%) Memory(30%)
freezed
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011 16

17. Agenda
•Setting the context…
- Background and Cgroups Definition
- Significance of Cgroups
• Cgroups: Deeper View
- Features and Design
- Cgroups Subsystems
- Cgroups Implementation
•Cgroups Use cases
- Usage Illustration
- Sample Use cases
•Cgroups : Challenges and Future
- Key Challenges
- Future
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011 17

18. Key Challenges
 Throughput intensive system, Cgroups may add
performance overhead
 It's not currently possible to bind a new subsystem to
an active cgroup hierarchy, or to unbind a subsystem
from an active cgroup hierarchy
 Though Cgroups acts like lighter virtualization, the
same can be applied only to native system
 Large multi-core systems, the linear performance
scalability is needed to utilise the resource better
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011 18

19. Future…
 Throughput optimization
 Virtualization Possibilities and Lower Level Resource
Abstraction
 Tuning and Development for Large Multi-core systems
 Multicore clustering support and Specialized
Subsystems
and …shaping as an inevitable component in linux kernel
for effective resource control and management…
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011 19

20. Ashish Pandey (ashish.pandey@huawei.com) Roopesh Keeppattu (roopesh@huawei.com)
Liuyang(liuyang@huawei.com) Sanil Kumar D. (sanil@huawei.com)
HUAWEI TECHNOLOGIES CO., LTD. OSI Days Nov 2011 20