Perfomance Tuning, Monitoring, Management: Getting the Most from SUSE Linux Enterprise Server

Performance Tuning,
Monitoring, Management
Getting the Most out of SUSE Linux Enterprise Server
®

Matthias G. Eckermann
Senior Product Manager
SUSE Linux Enterprise
mge@novell.com

Agenda

Performance Analysis and Tuning

Kernel Resource Management with Control Groups

Built-in Monitoring Capabilities

2 © Novell, Inc. All rights reserved.

Part I:
Performance Analysis and Tuning

General Considerations
(Hardware, Configuration,...)

Hardware and Configuration

Ultimately, hardware and its configuration set the
upper limits for our tuning efforts.
Are we starting with the best possible (minimum
needed) hardware platform and components?
– CPU speed only critical for compute-intense tasks
– RAM (amount and speed) and interconnects do matter
– Bottleneck I/O: network bandwidth, disk,...

Is the hardware configuration appropriate?

The weakest link kills performance!

(Hardware) Configuration

Optimize storage configuration
– Optimize distribution of data across controllers/disks
– Swap to extra disk
– Use RAID with striping
Tune hardware setup (BIOS, EFI,...)
– Only enable/proble what you have.
– Tune for fast reboot vs. startup checks (if desired)
– Carefully review all settings.
Disable unneeded services
# rc<SERVICE> stop


Where Has My Memory Gone!?

Slab Cache
– Structures of much less than one page in size
– Generic slabs of predefined sizes (32, 64) plus
slabs for specific data structures

Page Cache
– Pages with actual contents of files (or block device)
usually the largest, by far

Buffer Cache
– File system metadata


Identifying Problems

Start by finding the bottleneck: I/O, disk, mem,...
iostat to identify overloaded drives
– package syssat
#iostat -x 1

vmstat for basic system usage
# vmstat 1

r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 76804 8268 14996 167396 1 1 36 64 132 197 4 1 92 3
0 0 76804 8268 14996 167396 0 0 0 0 1023 879 3 0 97 0
0 0 76804 8300 14996 167396 0 0 0 0 1158 1134 2 0 98 0

Slabtop for slab cache use

Picking a File System

Pick the right file system for the task
– Indexed metadata

– File sizes

– Number of files

– Workloads (database, mail server,...)

– AccessPaths

– Dump/Restore


SUSE Linux Enterprise ®

Filesystems
Feature Ext 3 reiserfs XFS OCFS 2 btrfs
Data/Metadata Journaling •/• ○/• ○/• ○/• N/A [3]
Journal internal/external •/• •/• •/• •/○ N/A
Offline extend/shrink •/• •/• ○/○ •/○ •/•
Online extend/shrink •/○ •/○ •/○ •/○ •/•
Inode-Allocation-Map table u. B*-tree B+-tree table B-tree
Sparse Files • • • • •
Tail Packing ○ • ○ ○ •
Defrag ○ ○ • ○ •
ExtAttr / ACLs •/• •/• •/• •/• •/•
Quotas • • • • •
Dump/Restore • ○ • ○ ○
Blocksize default 4KiB
max. Filesystemsize [1] 16 TiB 16 TiB 8 EiB 16 TiB 16 EiB
max. Filesize [1] 2 TiB 1 EiB 8 EiB 1 EiB 16 EiB
Support Status SLES SLES SLES SLE HA Technology
Preview
SUSE® Linux Enterprise was the first enterprise Linux distribution to support journaling filesystems and logical volume managers back in 2000. Today, we have customers running XFS and ReiserFS with more
than 8TiB in one filesystem, and the SUSE Linux Enterprise engineering team is using our 3 major Linux journaling filesystems for all their servers. We are excited to add the OCFS2 cluster filesystem to the range of
supported filesystems in SUSE Linux Enterprise. For large-scale filesystems, for example for file serving (e.g., with with Samba, NFS, etc.), we recommend using XFS. (In this table "+" means "available/supported"; "-"
is "unsupported")
[1] The maximum file size above can be larger than the filesystem's actual size due to usage of sparse blocks. It should also be noted that unless a filesystem comes with large file support (LFS), the maximum file size on a 32-bit system is 2 GB
(231 bytes). Currently all of our standard filesystems (including ext3 and ReiserFS) have LFS, which gives a maximum file size of 263 bytes in theory. The numbers given in the above tables assume that the filesystems are using 4 KiB block size.
When using different block sizes, the results are different, but 4 KiB reflects the most common standard.

[2] In this document: 1024 Bytes = 1 KiB; 1024 KiB = 1 MiB; 1024 MiB = 1 GiB; 1024 GiB = 1 TiB; 1024 TiB = 1 PiB; 1024 PiB = 1 EiB (see also http://physics.nist.gov/cuu/Units/binary.html )
[3] Btrf s is a copy -on-write logging-sty le f ile sy stem, so rather than needing to journal changes bef ore writing them in-place, it writes them in a new location, and then links it in. Until the last write,
the new changes are not “committed.”


File Systems: ReiserFS

Applications that use many small files

– Mail servers

– NFS servers

– Database servers

or other applications that use synchronous I/O


File Systems: Ext3

Default file system in SUSE Linux Enterprise 11
®

Best suited for
– Small (<100GiB) file systems
When using Ext3 with many files in one directory,
consider enabling btree support
(enabled by default in SUSE Linux Enterprise
Server 11 SP 1)
# mkfs.ext3 -O dir_index
When using Ext3 with multiple threads appending to files
in the same directory, consider turning preallocation on
# mount -o reservation


File Systems: XFS

Best suited for
– Medium (>100GiB) to very large file systems (> 1 TiB)
– Large files/many files
– Streaming multimedia (low latencies)

Special features and capabilities
– dump/restore
– online filesystem-check
– online-defragmentation


Cluster File System: OCFS2

OCFS2 (Oracle Cluster File System)
• Shared access by multiple nodes
– Ensures data integrity in case of a node-failure
– Scale-out for data access
• Generic use
– POSIX-compliant
– Cluster-aware POSIX locking
• Higher throughput
– Parallel I/O
• Disaster Tolerance
– Integration with data replication for dual node


Filesystems: btrfs

• Integrated Volume Management
• Support for copy on write
• Powerful snapshot capabilities
• Scalability
• Data integrity (checksums)
• Full community support
• Technology preview in
SUSE Linux Enterprise Server 11 SP 1
®


Barriers

SUSE Linux Enterprise defaults to maximum data integrity
®

guarantee by enforcing barriers from the file system so
that reordering of journal writes cannot happen.
This may cost some performance; tunable via mount option
ReiserFS
– enable with “barrier=flush” (default)
– disable with “barrier=none”
Ext3
– enable with “barrier=1” (default)
– disable with “barrier=0”
XFS
– enable with “barrier”
– disable with “nobarrier”

Logging Modes

Journaling file systems offer different modes to write
the actual data
For ReiserFS and Ext3, mount option data=<X>
– data=ordered: use barriers for data
no risk exposing old data (default)
– data=writeback: no barriers for data
fastest in many workloads
– journal: use journal for data
generally slow, but can improve mail server workloads

By default, SUSE Linux Enterprise Server ensures
®

data integrity at the cost of some performance

Dedicated Logging Devices

ReiserFS
mkreiserfs -j /dev/xxx -s 8193 /dev/xxy
reiserfstune –journal-new-device /dev/xxx -s 8193

Ext3
mke2fs -O journal_dev /dev/xxx
mke2fs -j -J device=/dev/xxx,size=8193 /dev/xxy
tune2fs -J device=/dev/xxx,size=8193 /dev/xxy

XFS
mkfs.xfs -l logdev=/dev/xxx,size=10000b /dev/xxy


File System Tuning

Split file systems based on data access patterns

– Keep commit heavy data away from data that does not
have to be synchronous

– Keep streaming writes and reads on different spindles than
random I/O

Consider disabling atime updates on files and directories

# mount -o noatime,nodiratime


File System Tuning

Optimize directory layout for the file system

– Keep data that will be accessed together in the
same subdirectories

– Spread data out into different subdirectories to increase
large file concurrency

– Different file systems order directories differently


I/O Scheduler

Flexible, pluggable I/O scheduler
Selectable via boot parameter elevator=<X>
– noop
– deadline
– as (default in mainline kernels)
– cfq (default in SUSE Linux Enterprise)
®

I/O Scheduler per device
– Check
/sys/block/*DEV*/queue/iosched
– Set
echo SCHEDNAME > /sys/block/*DEV*/queue/scheduler


I/O Scheduler: Noop

No reordering, just merging

Best for storage with extensive caching and
scheduling of its own, such as:

MultiPathing

Activated by boot parameter elevator=noop


I/O Scheduler: Deadline

Per-request service deadline

– Caps maximum latency per request

– Maintains good disk throughput

Best for disk-intensive database applications

Activated by boot parameter elevator=deadline


I/O Scheduler: Anticipatory

Similar to “deadline”, but anticipates reads by putting
them in front of the queue and delays a few ms after
every read
– Maximizes throughput
– At the cost of increasing latency
Best for file servers and desktop workloads with
single IDE/SATA disks.
Default in mainline kernels

Activated by boot parameter elevator=as


I/O Scheduler: CFQ

Complete Fair Queuing
Treat all competing processes equally by keeping
a unique request queue for each and giving equal
bandwidth to each queue
– Good compromise between throughput and latency
– Minimal worst case latency on all reads and writes
Suitable for a wide variety of applications
Default in SUSE Linux Enterprise ®

Activated by boot parameter elevator=cfq


Block Layer Tuning

Spreading the load across controllers

– Per-target locking for SCSI

– Software RAID bandwidth

Battery backed caching


Blocker Layer Tunables

Block read ahead buffer
/sys/block/<sdX/hdX>/queue/read_ahead_kb
Default is 128. Increase to 512 for fast storage
(SCSI disks or RAID)
May speed up streaming reads a lot

Number of requests
/sys/block/<sdX/hdX>/queue/nr_requests
Default is 128. Increase to 256 with CFQ
scheduler for fast storage
Increases throughput at minor latency expense

Buffer Flushing

How to write dirty pages to disk
This can be tuned by
– /proc/sys/vm/dirty_ratio (40%)
Generator of dirty data starts writeback.
– /proc/sys/vm/dirty_background_ratio (10%)
– /proc/sys/vm/dirty_expire_centisecs (3000)
How long may dirty pages remain dirty?
– /proc/sys/vm/dirty_writeback_centisecs (500)
How often does bdflush wake up?
Defaults are pretty high which is good for databases
(but may result in lots of unreclaimable pagecache)
For other workloads (HPC) you may want to lower these

VM: Swapiness

The threshold when processes should be swapped
can be tuned via
– /proc/sys/vm/swappiness

Default is 60, which works well if you want to swap out
daemons or programs which have not done a lot lately

Higher values will provide more buffer/page cache,
lower values will wait longer to swap out idle processes


NUMA (1)

NUMA = Non-uniform Memory Architecture
SUSE Linux Enterprise detects and uses NUMA
®

topology and automatically
– Prefers memory that is local to a node;

– Evenly balances system data across nodes; NUMA

– Gracefully handle CPU-less nodes; etc.

Also applications can (and should) optimize for
NUMA topology!


NUMA (2)

The NUMA system can be tuned via `numactl $CMD`;
the settings then apply to $CMD and all of its children
– --preferred=255

– --membind=!0-1
NUMA
– --cpunodebind=2-5

– --physcpubind=2-5

– --localalloc (always allocate from current node)

Node 0 may be the most contended, so avoid it

Miscellaneous
(Scheduler, Network)

Binding Processes/Interrupts to CPUs

Problem: context switching costs
CPU affinity: binding CPUs to a specific process
can improve performance
– taskset 0x3 [-p pid] [command]

In this example, 0x3 is a bitmap referring to CPUs 1
and 2; 0x6 would be CPUs 2 and 4.
Bind interrupts to CPUs
– cat /proc/interrupts

– echo 0x3 > /proc/irq/0/smp_affinity

– Example: distribute NICs among CPUs.


Network Improvements

Gigabit Ethernet and 10g
– Significant interrupt overhead reduction

– Consider Jumbo Frames (larger 1500 bytes)

– # ifconfig <DEV> mtu 9000

NFS modes
– TCP (default) vs UDP

– NFSv3 (default) vs NFSv4

– rsize=<X>/wsize=<X>
- read/write in chunks of <X> bytes
- default is 1024, use 8192 for higher throughput

Async I/O, O_DIRECT

Asynchronous I/O
– Specific model for concurrency
– Heavily used by databases

Direct I/O (O_DIRECT) on block devices or files
– Databases like to use raw disks. Historically /dev/raw
was used, but O_DIRECT is more performant.
– Files should be preallocated (no holes, no appending);
the system falls back to buffered I/O otherwise.
– In both cases: cache pollution benefits
– Not specific to database workloads!


Part II:
Kernel Resource Managemet with
Control Groups

Control Groups

• Understanding control groups: An in-depth overview

– What Control Groups is designed to do

– How Control Groups work

• Using control groups in
SUSE Linux Enterprise Server 11
®

– Understanding the components


Understanding Control Groups
An In-depth Overview

What Are Control Groups?

Control Groups provide a mechanism for
aggregating/partitioning sets of tasks, and all their
future children, into hierarchical groups with
specialized behavior.
– cgroup is another name for Control Groups
– Partition tasks (processes) into a one or many groups of
tree hierarchies
– Associate a set of tasks in a group to a set subsystem
parameters
– Subsystems provide the parameters that can be assigned
– Tasks are affected by the assigning parameters


Example of the Capabilities of a cgroup

Consider a large university server with various users -
students, professors, system tasks etc. The resource
planning for this server could be along the following lines:

CPUs Memory Network I/O
Top cpuset (20%) Professors = 50% WWW browsing = 20%
/ Students = 30% /
CPUSet1 CPUSet2 System = 20% Prof (15%) Students (5%)
| |
(Profs) (Students) Disk I/O Network File System (60%)
60% 20% Professors = 50%
Students = 30% Others (20%)
System = 20%

45 © Novell, Inc. All rights reserved. Source: /usr/src/linux/Documentation/cgroups/cgroups.txt
•

Control Group Subsystems

Two types of subsystems
• Isolation and special controls
– cpuset, namespace, freezer, device, checkpoint/restart
• Resource control
– cpu(scheduler), memory, disk i/o, network

Each subsystem can be mounted independently
– mount -t cgroup -o cpu none /cpu
– mount -t cgroup -o cpuset none /cpuset
or all at once
– mount -t cgroup none /cgroup
46 © Novell, Inc. All rights reserved. Source: http://jp.linuxfoundation.org/jp_uploads/seminar20081119/CgroupMemcgMaster.pdf
•

Cpuset Subsystem (Isolation)

Cpuset is for tying processes to cpu and memory.

Process Process Process Process Process Process
Group A1 Group A2 Group B Group A1 Group A2 Group B

Memory Memory Memory Memory

•

Namespace Subsystem (Isolation)

Namespace is for showing private view of system to

processes in cgroup. Mainly used for OS-level

virtualization. This subsystem itself has no special

functions and just tracks changes in namespace.

•

Freezer Subsystem (Control)

Freezer cgroup is for freezing (stopping) all tasks in
a group.

mount -t cgroup none /freezer -o freezer

....put task into /freezer/tasks...

echo FROZEN > /freezer/freezer.state

echo RUNNING > /freezer/freezer.state

•

Device Subsystem (Isolation)

A system administrator can provide a list of devices
that can be accessed by processes under cgroup
– Allow/Deny Rule

– Allow/Deny : READ/WRITE/MKNOD

Limits access to device or file system on a device to
only tasks in specified cgroup

•

Checkpoint/Restart Subsystem
(Control)
• Save all process's status in a cgroup to a dump file,
restart it later (or just save state and continue)

• For allowing “saved container” moved between
physical machines (as VM can do)

• Dump all process's image to a file

•

CPU Subsystem (Resource Control)

• Share CPU bandwidth between groups by group
scheduling function of CFS (the scheduler)
• Mechanically complicated

Share = 2000 Share = 1000 Share = 4000

Memory Subsystem
(Resource Control)
• For limiting memory usage of user space processes.

• Limit LRU (Least Recently Used) pages

– Anonymous and file cache

• No limits for kernel memory

– Maybe in another subsystem if needed

•

Disk I/O Subsystem
(Resource Control) (Draft)
• 3 proposals are currently being discussed
– dm-ioband, io-throttle, io-controller
• Consensus has not been reached but io-controller
seems to taking the lead
– Both dm-ioband and io-throttle suffer from a significant
problem: they can defeat the policies (such as I/O priority)
being implemented by the I/O scheduler.
– Io-throttle is does bandwidth control at the I/O scheduler level
– Designed to work with mainline I/O controllers: CFQ, deadline,
Anticipatory, and no-op but requires significant changes
– Currently v4 as of June 8, 2009 and based on
2.6.30-rc8 kernel
Source: http://jp.linuxfoundation.org/jp_uploads/seminar20081119/CgroupMemcgMaster.pdf
Source: http://lwn.net/Articles/331857/
Source: http://lwn.net/Articles/332839/
54 © Novell, Inc. All rights reserved. Source: http://lkml.org/lkml/2009/6/8/580

Network I/O Subsystem
(Resource Control) (Draft)
• Like the Disk I/O subsystem, it seems the jury is
still out on the implementation of this subsystem

• Kernel developers are talking about traffic control

– cgroup_tc - This patch provides a simple resource
controller which uses traffic control (tc) features already
in the Linux kernel

– Not much discussion on this topic since late 2008

Source: http://lkml.org/lkml/2008/7/22/361
Source: https://lists.linux-foundation.org/pipermail/containers/2008-August/012419.html
55 © Novell, Inc. All rights reserved. Source: https://lists.linux-foundation.org/pipermail/containers/2008-August/012512.html

Reading More on cgroups

Remember to install kernel source!!
– /usr/src/linux/Documentation/cgroups/cgroup.txt
– /usr/src/linux/Documentation/cpusets.txt
– /usr/src/linux/Documentation/controllers/*
– /usr/src/linux/Documentation/scheduler/sched-design-CFS.txt
– /usr/src/linux/Documentation/kernel-parameters.txt

Additional RPM packages
– libcgroup1 - /usr/share/doc/packages/libcgroup1/README*
– cpuset (Alex Tsariounov) -
/usr/share/doc/packages/cpuset/cset*.txt


Reading More on cgroups (continued)

Manpages
– man cpuset

– man cset

On the web
– http://lkml.org/lkml/2009/2/9/372



– http://kerneltrap.org/mailarchive/linux-kernel/2008/6/18/2161114/thread


Using Control Groups in
®

Preparing
®

• Start with patched SLES11 install

• Add the following packages
– libcgroup1

– cpuset

– libcpuset1

– kernel-source (Documentation purposes)

– gcc (Needed to compile the stress tool)


What Subsystems Are Available?

A way to figure this out.
– mount -t cgroup none /cgroup
– cat /proc/mounts

Current subsystems in SUSE Linux Enterprise
Server 11
– rw,freezer,devices,cpuacct,cpu,ns,cpuset
– memory – Disabled by default
> Add a kernel parameter - cgroup_enable=memory

Possible future subsystems in SLES 11
– Disk and Network subsystem controllers


Generating Load on
®

Search the Web for “linux load generator” - Results:
• http://devin.com/lookbusy/
• http://www.ibm.com/developerworks/linux/library/l-stress/index.html
– Good article
• http://ltp.sourceforge.net/
– Powerful toolkit for Linux developers
• http://hardware.slashdot.org/article.pl?sid=05/04/06/218233
– Simple scripting examples
• http://weather.ou.edu/~apw/projects/stress/
– Probably best choice
– Available (community driven) also at:

http://software.opensuse.org/search?baseproject=SUSE:SLE-11&p=1&q=stress


Crash Course on CPUSETs
The Hard Way

• Determine the number of CPUs and Memory Nodes
– Look at /proc/cpuinfo and /proc/zoneinfo
• Creating the CPUSET hierarchy
mkdir /dev/cpuset
mount -t cpuset cpuset /dev/cpuset
cd /dev/cpuset
mkdir Charlie
cd Charlie
/bin/echo 2-3 > cpus
/bin/echo 1 > mems
/bin/echo $$ > tasks
# The current shell is now running in cpuset Charlie
# The next line should display '/Charlie'
cat /proc/self/cpuset
• Removing the CPUSET
cat /dev/cpuset/Charlie/tasks (move any remaining tasks!!)
rmdir /dev/cpuset/Charlie


Crash Course on CPUSETs
The Easy Way – Thanks to Alex Tsariounov of Novell ®

• Determine the number of CPUs and Memory Nodes
– cset set --list
• Creating the CPUSET hierarchy
– cset set --cpu=2-3 --mem=1 --set=Charlie
• Starting processes in a CPUSET
– cset proc --set Charlie --exec -- stress -c 1 &
• Moving existing processes to a CPUSET
– cset proc --move --pid PID --toset=Charlie
• List task in a CPUSET
– cset proc --list --set Charlie
• Removing a CPUSET
– cset set --destroy Charlie


Follow It Up with cgroups
The Hard way

• Creating the cgroup hierarchy
mkdir /dev/cgroup
mount -t cgroup cgroup /dev/cgroup
cd /dev/cgroup
mkdir priority
cd priority
cat cpu.shares

• Understanding cpu.shares
– 1024 is the default (more in sched-design-CFS.txt) = 50% utilization
– 1524 = 60% utilization
• Changing cpu.shares
– /bin/echo 1024 > cpu.shares


More cgroup Functionality to Learn

The libcgroup1 package
• Basic tools in user space to simplify resource
management functionality
– uid, gid or exec rules for placement of a task
– /etc/init.d/cgconfig – setup cgroup filesystem based on
/etc/cgconfig.conf
• UID/GID rules
– Managed in /etc/cgrules.conf by root user
• EXEC rules
– Fully managed by a user in a config file in their home directory
• Methods used to place task in proper cgroup
– pam_cgroup (at login); cgexec (task start); cgclassify (task move)
– User space daemon (cgred in /etc/init.d and /etc/sysconfig)

Linux Containers
LXC

• Build upon CGroups and specific kernel settings;
use “lxc-checkconfig” to check compliance
• Fully enabled in SUSE Linux Enterprise Server 11 SP1
• Basic Functionality
lxc-execute --name=NAME -- COMMAND
• Function Overview
– lxc-start lxc-execute / lxc-stop
– lxc-freeze lxc-unfreeze
– Monitoring: lxc-ps, lxc-info, lxc-netstat, lxc-monitor
– Modifying CGroup parameters: lxc-cgroup


Part III:
Built-in Monitoring Capabilities

Monitoring Overview and Hands-on

• Low Level
– smartmontools - Monitor for S.M.A.R.T. Disks and Devices
– sensors - Hardware health monitoring for Linux
– iptraf - TCP/IP Network Monitor
– pcp - Performance Co-Pilot
(system-level performance monitoring)
– sysstat - Sar and Iostat Commands for Linux
– perfmon
– blktrace, ltrace, strace
– systemtap - Instrumentation System
• High Level
– argus – network auditing tool
– nagios


Unpublished Work of Novell, Inc. All Rights Reserved.
This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc.
Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope
of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified,
translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc.
Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.

General Disclaimer
This document is not to be construed as a promise by any participating company to develop, deliver, or market a
product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in
making purchasing decisions. Novell, Inc. makes no representations or warranties with respect to the contents
of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any
particular purpose. The development, release, and timing of features or functionality described for Novell products
remains at the sole discretion of Novell. Further, Novell, Inc. reserves the right to revise this document and to
make changes to its content, at any time, without obligation to notify any person or entity of such revisions or
changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc.
in the United States and other countries. All third-party trademarks are the property of their respective owners.

Perfomance Tuning, Monitoring, Management: Getting the Most from SUSE Linux Enterprise Server

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (9)

Similar to Perfomance Tuning, Monitoring, Management: Getting the Most from SUSE Linux Enterprise Server

Similar to Perfomance Tuning, Monitoring, Management: Getting the Most from SUSE Linux Enterprise Server (20)

More from Novell

More from Novell (20)

Perfomance Tuning, Monitoring, Management: Getting the Most from SUSE Linux Enterprise Server