1. Linux Performance Analysis
Will Sterling
Linux & UNIX Consultant
@
PARSEC Group

2. Agenda
Performance Analysis Tools
•
Performance Profiling Tools
•
Identifying Common Performance Issues
•
Linux Performance Analysis

3. Performance Analysis Tools
CPU Memory Process Network
top top ps nettop
mpstat vmstat nmon sar
ps ps Disk ethtool
sar sar nmon nmon
/proc Free Sar
nmon /proc Iostat
nmon vmstat
Linux Performance Analysis

4. Top
top ­ 01:45:25 up 12 min, 1 user, load average: 12.33, 7.84, 3.42
Tasks: 141 total, 14 running, 117 sleeping, 0 stopped, 10 zombie
Cpu(s): 93.4%us, 6.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8191996k total, 267248k used, 7924748k free, 10532k buffers
Swap: 0k total, 0k used, 0k free, 106008k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23202 root 20 0 13060 992 620 R 12.3 0.0 0:00.37 bc
23210 root 20 0 13060 992 620 R 12.3 0.0 0:00.37 bc
23218 root 20 0 13060 992 620 R 9.6 0.0 0:00.29 bc
23226 root 20 0 13060 992 620 R 9.6 0.0 0:00.29 bc
23234 root 20 0 13060 992 620 R 8.0 0.0 0:00.24 bc
23242 root 20 0 13060 944 620 R 6.3 0.0 0:00.19 bc
23250 root 20 0 13060 928 620 R 4.3 0.0 0:00.13 bc
23258 root 20 0 13060 892 620 R 3.3 0.0 0:00.10 bc
2519 root 20 0 104m 1652 1272 S 2.0 0.0 0:05.85 bc.ksh
23266 root 20 0 12928 860 620 R 1.7 0.0 0:00.05 bc
23274 root 20 0 12928 852 620 R 1.3 0.0 0:00.04 bc
1 root 20 0 19400 1508 1192 S 0.0 0.0 0:01.20 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:00.02 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.07 migration/0
6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
Linux Performance Analysis

5. Top Cont.
top ­ 01:59:43 up 26 min, 1 user, load average: 0.63, 2.83, 4.08
Tasks: 121 total, 3 running, 118 sleeping, 0 stopped, 0 zombie
Cpu(s): 43.3%us, 8.7%sy, 0.0%ni, 48.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8191996k total, 266016k used, 7925980k free, 11944k buffers
Swap: 0k total, 0k used, 0k free, 108088k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24222 root 20 0 103m 928 780 R 98.8 0.0 0:03.16 awk
17758 root 20 0 104m 1572 1264 S 7.9 0.0 0:40.30 bc.ksh
1 root 20 0 19400 1508 1192 S 0.0 0.0 0:01.54 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:00.83 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.07 migration/0
6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
7 root RT 0 0 0 0 S 0.0 0.0 0:00.76 migration/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.06 migration/1
9 root 20 0 0 0 0 S 0.0 0.0 0:00.03 ksoftirqd/1
10 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
11 root RT 0 0 0 0 S 0.0 0.0 0:00.80 migration/2
12 root RT 0 0 0 0 S 0.0 0.0 0:00.02 migration/2
13 root 20 0 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/2
14 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/2
15 root RT 0 0 0 0 S 0.0 0.0 0:00.80 migration/3
Linux Performance Analysis

6. mpstat
[root@server­­­62012­04­29­083219­0000 ~]# mpstat ­P ALL 5 2
Linux 2.6.32­220.2.1.el6.x86_64 (server­­­62012­04­29­083219­0000.localdomain)
04/29/2012 _x86_64_ (4 CPU)
02:24:22 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest
%idle
02:24:27 AM all 26.32 0.00 0.05 0.00 0.00 0.00 0.00 0.00
73.63
02:24:27 AM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
100.00
02:24:27 AM 1 0.00 0.00 0.20 0.00 0.00 0.00 0.00 0.00
99.80
02:24:27 AM 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
100.00
02:24:27 AM 3 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00
Linux Performance Analysis

7. mpstat
[root@server­­­62012­04­29­083219­0000 ~]# mpstat ­P ALL 5 2
Linux 2.6.32­220.2.1.el6.x86_64 (server­­­62012­04­29­083219­0000.localdomain)
04/29/2012 _x86_64_ (4 CPU)
02:24:22 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest
%idle
02:24:27 AM all 26.32 0.00 0.05 0.00 0.00 0.00 0.00 0.00
73.63
02:24:27 AM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
100.00
02:24:27 AM 1 0.00 0.00 0.20 0.00 0.00 0.00 0.00 0.00
99.80
02:24:27 AM 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
100.00
02:24:27 AM 3 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00
Linux Performance Analysis

8. ps aux
[root@server­­­62012­04­29­083219­0000 ~]# ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 19400 1508 ? Ss 01:33 0:01 /sbin/init
root 2 0.0 0.0 0 0 ? S 01:33 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S 01:33 0:01 [migration/0]
root 4 0.0 0.0 0 0 ? S 01:33 0:00 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S 01:33 0:00 [migration/0]
root 6 0.0 0.0 0 0 ? S 01:33 0:00 [watchdog/0]
­­snip­­
root 1295 0.0 0.0 97816 3836 ? S 01:34 0:00 sshd: root@pts/
root 1298 0.0 0.0 108356 1908 pts/0 Ss 01:34 0:00 ­bash
root 7420 0.0 0.0 17336 588 ? Ss 02:01 0:00 /usr/sbin/anacr
root 24143 0.0 0.0 110396 1148 pts/0 R+ 02:33 0:00 ps aux
root 24138 88.5 0.0 105996 928 pts/0 R 02:32 0:12 awk BEGIN {for(
Linux Performance Analysis

9. Sar: CPU
[root@server­­­62012­04­29­083219­0000 ~]# sar ­u
Linux 2.6.32­220.2.1.el6.x86_64 (centos) 04/29/2012 _x86_64_ (4 CPU)
01:33:21 PM LINUX RESTART
01:40:01 PM CPU %user %nice %system %iowait %steal %idle
01:50:01 PM all 83.58 0.00 5.92 0.00 0.00 10.50
02:00:01 PM all 30.32 0.00 18.20 0.01 0.00 51.48
02:10:01 PM all 7.97 0.00 15.11 0.02 0.00 76.90
02:20:01 PM all 1.55 0.00 0.02 0.00 0.00 98.42
02:30:01 PM all 1.61 0.00 0.03 0.00 0.00 98.36
02:40:01 PM all 1.83 0.00 0.03 0.00 0.00 98.14
Average: all 21.24 0.00 6.19 0.00 0.00 72.56
Linux Performance Analysis

10. SAR: CPU
[root@server­­­62012­04­29­083219­0000 ~]# sar ­u
Linux 2.6.32­220.2.1.el6.x86_64 (centos) 04/29/2012 _x86_64_ (4 CPU)
01:33:21 PM LINUX RESTART
01:40:01 PM CPU %user %nice %system %iowait %steal %idle
01:50:01 PM all 83.58 0.00 5.92 0.00 0.00 10.50
02:00:01 PM all 30.32 0.00 18.20 0.01 0.00 51.48
02:10:01 PM all 7.97 0.00 15.11 0.02 0.00 76.90
02:20:01 PM all 1.55 0.00 0.02 0.00 0.00 98.42
02:30:01 PM all 1.61 0.00 0.03 0.00 0.00 98.36
02:40:01 PM all 1.83 0.00 0.03 0.00 0.00 98.14
Average: all 21.24 0.00 6.19 0.00 0.00 72.56
Linux Performance Analysis

11. SAR: Memory
[root@server­­­62012­04­29­083219­0000 ~]# sar ­r
Linux 2.6.32­220.2.1.el6.x86_64 (centos) 04/29/2012 _x86_64_ (4 CPU)
01:33:21 PM LINUX RESTART
01:40:01 PM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit
01:50:01 PM 7930632 261364 3.19 10940 106008 64096 0.78
02:00:01 PM 7926756 265240 3.24 11976 108088 66280 0.81
02:10:01 PM 7926956 265040 3.24 12696 108488 64652 0.79
02:20:01 PM 7926700 265296 3.24 12824 108580 64652 0.79
02:30:01 PM 7925568 266428 3.25 12980 108624 65824 0.80
02:40:01 PM 7926460 265536 3.24 13076 108628 64652 0.79
02:50:01 PM 7925964 266032 3.25 13196 108776 64652 0.79
03:00:01 PM 7925964 266032 3.25 13344 108788 64652 0.79
03:10:01 PM 7925732 266264 3.25 13480 108832 64660 0.79
03:20:01 PM 7913020 278976 3.41 13608 109444 80496 0.98
Average: 7925375 266621 3.25 12812 108426 66462 0.81
Linux Performance Analysis

12. SAR: Disk
[root@server­­­62012­04­29­083219­0000 ~]# sar ­d
Linux 2.6.32­220.2.1.el6.x86_64 (centos) 04/29/2012 _x86_64_ (4 CPU)
01:33:21 PM LINUX RESTART
01:40:01 PM DEV tps rd_sec/s wr_sec/s avgrq­sz avgqu­sz await svctm %util
01:50:01 PM dev252­0 0.47 0.75 12.86 28.91 0.00 1.01 0.94 0.04
01:50:01 PM dev252­16 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:00:01 PM dev252­0 0.74 7.79 16.17 32.30 0.00 0.73 0.50 0.04
­­snip­­
[root@server­­­62012­04­29­083219­0000 ~]# ls ­l /dev/vd*
brw­rw­­­­ 1 root disk 252, 0 Apr 29 01:33 /dev/vda
brw­rw­­­­ 1 root disk 252, 16 Apr 29 01:33 /dev/vdb
Linux Performance Analysis

13. Sar: Network
[root@server­­­62012­04­29­083219­0000 ~]# sar ­n DEV
Linux 2.6.32­220.2.1.el6.x86_64 (centos) 04/29/2012 _x86_64_ (4 CPU)
01:33:21 PM LINUX RESTART
01:40:01 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
01:50:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
01:50:01 PM eth0 1.19 1.02 0.08 0.58 0.00 0.00 0.00
02:00:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:00:01 PM eth0 1.30 0.83 0.10 0.27 0.00 0.00 0.00
02:10:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:10:01 PM eth0 1.47 1.16 0.11 0.18 0.00 0.00 0.00
02:20:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:20:01 PM eth0 0.40 0.20 0.03 0.04 0.00 0.00 0.00
02:30:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:30:01 PM eth0 0.80 0.53 0.07 0.16 0.00 0.00 0.00
02:40:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00
02:40:01 PM eth0 0.64 0.42 0.05 0.27 0.00 0.00 0.00
Linux Performance Analysis

14. /proc: CPU
[root@server­­­62012­04­29­083219­0000 proc]# cat cpuinfo
processor : 0
­­snip­­
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 2
model name : QEMU Virtual CPU version 0.14.0
stepping : 3
cpu MHz : 2666.760
cache size : 4096 KB
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pse36 clflush
mmx fxsr sse sse2 syscall nx lm rep_good unfair_spinlock pni cx16 popcnt
hypervisor lahf_lm
bogomips : 5333.52
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
Linux Performance Analysis

15. /proc: /CPU
[root@server­­­62012­04­29­083219­0000 proc]# cat stat
cpu 295757 0 86819 1581480 160 49 190 0 0
cpu0 70616 0 22135 398368 47 40 55 0 0
cpu1 74539 0 21982 394194 23 0 47 0 0
cpu2 70228 0 21854 399389 42 6 46 0 0
Cpu3 80372 0 20846 389528 46 2 40 0 0
Total cycles spent: USER NICE SYSTEM IDLE IOWAIT
Linux Performance Analysis

16. nmon: CPU
http://nmon.sourceforge.net/pmwiki.php
Linux Performance Analysis

17. Nmon: Memory/Disk/Network
Linux Performance Analysis

18. ksar
Linux Performance Analysis

19. Common Performance Issues
Network Bottle Neck to NAS
•
Slow IO to Virtual Drives
•
Memory Paging
•
CPU
•
Processes Waiting on Upstream Applications
•
Miss-configured Storage
•
Linux Performance Analysis