AIX Performance Tuning Session at STU2017

1
IBM Systems Technical Events | ibm.com/training/events
© Copyright IBM Corporation 2017. Technical University/Symposia materials may not
be reproduced in whole or in part without the prior written permission of IBM.
AIX Performance Tuning
Paulo Sergio Lemes Queiroz
Systems Lab Services Servers Consultant
IBM

2
Session objectives
• Understand server behavior
●
Understand CPU utilization
• Monitoring CPU utilization
●
Lparstat / Vmstat / Mpstat
●
ps
●
Tunables

3
Understand server behavior
AIX implement a unified work model, of virtual resources accessed by the applications.
Therefore IO / Memory / IO systems will influence on each other.
●
Slow IO system will saturated memory which will saturate the processors on keep this
memory organized.
●
What if we have idle cpu and the system still slow ?
●
Poor configurated core pools can diminish virtual processor performance
●
Can the slowdown be caused by external factors ?
●
Too fast IO will saturate the processor that won't have time to work on the application
demand
●
What if the system think it's fast and in fact it isn't ?!
CPUCPU
IOIOMemoryMemory

4
IBM POWER processor work under the premisse that most of applications make use of at
least two threads that will share memory resources.
●
It support up to 8 parallel threads that will share memory region
●
These instructions will work at same region but never at the same address
●
More non-conflicting threads, better performance ( due cycle utilization
●
Fewer cycles to complete the process
What are threads ?
●
Threads are instructions blocks that will be executed at the processor.
●
There are two types of threads
●
Hard(heavy) thread → Those are the main execution threads of the process, and are
identified at the process itself at the “ps” command
●
Light threads → sub tasks called by the main/heavy thread

5
It's important to identify the amount of executing threads and it's behavior in order to
optimize IBM SMT setup to match workload.
Miss matched setup will lead to performance slowdown
How can this be identified ?

6
The simplest way to identify the amount of threads being executed at the server is through
the “ps” command.
“ps -Afmo pid,ppid,tid,thcount,flags,pri,cpu,bnd,sched,nice,dpgsz,spgsz,tpgsz,shmpgsz,time,etime,vsz,st,rssize,pmem,pcpu,args”

7
What if is there a process blocking access of these threads to the processor ?!
Still at ps command, we can validate this data:
“ps -guww”

8
Mapping thread behavior allow better setup of the server scheduler, which result into faster
processor allocation, improving performance.
This information is crucial to define the right SMT and throughput level of the server
The command smtctl allow setup different SMT levels ( 1,2,4,8 )
# smtctl
This system is SMT capable.
This system supports up to 4 SMT threads per processor.
SMT is currently enabled.
SMT boot mode is set to enabled.
SMT threads are bound to the same virtual processor.
proc0 has 4 SMT threads.
Bind processor 0 is bound with proc0
# schedo -aF | grep -i throu
vpm_throughput_mode = 0
vpm_fold_policy = 2
# smtctl
This system is SMT capable.
This system supports up to 4 SMT threads per processor.
SMT is currently enabled.
SMT boot mode is set to enabled.
SMT threads are bound to the same virtual processor.
proc0 has 4 SMT threads.
The vpm_throughput_mode setting at schedo
define how many threads will be dispatched to
the same processor before enable another
processor
The vpm_fold_policy define that unused
procesors should be disabled in order to allow
usage by other lpars and improve cache
affinity of running proccess

9
Generally applications that work mainly with light/soft threads benefit of higher
smt/throughput levels due better cache affinity
Workloads that generally work better with SMT8:
●
DB2 ( 9.5+ )
●
Mariadb ( mysql )
●
Oracle12c ( with thread model enabled )
●
Java ( when using mem-prefetch )
●
Nodejs
Workloads that generally work better with SMT4 ( Power8 ):
●
Oracle11g/12c ( default config )
●
Postgresql
●
SAP Applications
Workloads that work generally better with SMT2- ( Power7+ )
●
Nodejs
●
Redis
●
Progress
There is no magic setup

10
Assuming SMT tuning have been done, how can be determine if the server is
CPU starved ?
Standard tools show CPU and Core behavior and can be used to determine
resources shortage
●
Lparstat
●
Mpstat
●
vmstat

11
Lparstat
# Lparstat -h 5
System configuration: type=Shared mode=Uncapped smt=4 lcpu=8 mem=35840MB psize=16 ent=1.00
%user %sys %wait %idle physc %entc lbusy vcsw phint %hypv hcalls
----- ----- ------ ------ ----- ----- ------ ----- ----- ------ ------
49.1 16.7 0.0 34.2 1.25 125.0 16.7 1241 1 43.0 257053
46.4 17.5 0.0 36.1 1.14 114.4 14.6 1492 2 46.5 261770
47.8 17.7 0.0 34.5 1.09 109.1 14.5 876 1 46.4 252235
45.6 19.0 0.0 35.4 1.14 113.6 14.7 3255 3 47.9 264814
46.8 18.0 0.1 35.2 1.11 111.4 14.5 1386 4 45.1 254430
47.5 17.5 0.0 35.0 1.09 109.4 14.4 1303 2 43.1 261104
46.6 20.6 0.0 32.8 1.42 141.6 20.5 2071 6 44.2 348788
45.3 19.0 0.0 35.7 1.19 119.1 15.3 3031 1 45.6 268902
46.2 17.8 0.0 36.0 1.15 115.0 14.9 2671 2 55.4 258718
47.2 17.3 0.0 35.5 1.10 109.5 14.0 837 1 48.1 257845
45.6 17.9 0.0 36.5 1.13 113.1 14.4 2192 2 44.2 264767
46.1 18.3 0.0 35.6 1.14 113.8 14.5 2053 3 42.6 255954
50.5 21.0 0.0 28.5 1.46 145.9 22.8 1147 5 45.1 348883
46.9 17.2 0.0 35.9 1.17 116.8 15.0 849 2 43.9 254564
46.2 17.9 0.0 35.9 1.13 113.4 14.6 3259 1 46.5 265825
46.6 19.1 0.0 34.3 1.11 111.1 15.0 2391 2 44.1 255031
45.7 18.1 0.0 36.1 1.15 114.6 14.5 2528 6 45.2 266687
17.3 14.0 0.0 68.7 0.52 52.3 6.3 4464 1 24.5 93596
3.6 7.3 0.0 89.1 0.21 20.8 2.0 4528 0 12.5 14239

12
Lparstat

If the ilcs ( involuntary logical context switch ) increase too much above the vlcs ( voluntaty context
switch ) this mean that AIX is being ejected from the core by the Hypervisor ( stop using the core )

If cs vs ics ratio increase above 25% this indicate processor starvation

If ilcs vs vlcs increase above 25% this indicate core starvation ( maybe the server has too many virtual
CPUs )

As much as data is kept under S3hrd better, because the data was into the local cache lines.

As much as possible S5hrd must be avoided, because that means that the thread had to be migrated to
another core ( specially S5rd that means another CEC )

13
vmstat
Vmstat -Iwt 5
System configuration: lcpu=14 mem=266240MB ent=7.00
kthr memory page faults cpu time
----------- --------------------- ------------------------------------ ------------------ ----------------------- --------
r b p avm fre fi fo pi po fr sr in sy cs us sy id wa pc ec hr mi se
4 0 0 21195441 34161325 0 256 0 0 0 0 8399 328947 34752 21 12 65 2 3.05 43.6 13:02:32
7 0 0 21189285 34166734 0 128 0 0 0 0 8159 986241 149309 28 19 53 0 4.25 60.7 13:02:37
3 0 0 21184260 34170995 0 134 0 0 0 0 8070 671558 122153 20 15 65 0 3.26 46.5 13:02:42
5 0 0 21204688 34149780 0 627 0 0 0 0 7908 831248 137840 21 15 64 0 3.39 48.4 13:02:47
3 0 0 21184538 34169598 0 51 0 0 0 0 6703 757693 110757 21 14 65 0 3.22 46.0 13:02:52
6 0 0 21184461 34169171 5 96 0 0 0 0 4843 669222 93760 21 14 65 0 3.19 45.6 13:02:57
5 0 0 21191227 34161599 0 230 0 0 0 0 7037 1042299 163814 26 20 54 0 4.14 59.1 13:03:02
7 0 0 21189396 34162699 0 589 0 0 0 0 4176 842586 108623 26 17 57 0 3.82 54.5 13:03:07
5 0 0 21217790 34133531 0 597 0 0 0 0 6319 917415 124393 27 17 56 0 3.97 56.7 13:03:12
5 0 0 21190929 34159552 10 538 0 0 0 0 5709 1047379 126508 28 19 53 1 4.15 59.3 13:03:17
3 0 0 21217596 34132482 52 60 0 0 0 0 5260 563976 65521 20 12 68 0 2.97 42.5 13:03:22
6 0 0 21201050 34147958 41 172 0 0 0 0 4423 913617 138471 21 14 64 0 3.28 46.9 13:03:27
6 0 0 21216338 34131837 8 251 0 0 0 0 4385 747083 94672 21 14 65 0 3.23 46.2 13:03:32
5 0 0 21196876 34150304 0 181 0 0 0 0 5543 715469 116953 20 14 66 0 3.12 44.5 13:03:37
7 0 0 21195473 34151356 0 70 0 0 0 0 6266 918217 126810 23 16 61 0 3.61 51.6 13:03:42
5 0 0 21195364 34150991 0 373 0 0 0 0 5283 590105 81715 19 12 69 0 2.87 41.0 13:03:47
3 0 0 21195307 34150303 0 139 0 0 0 0 4302 976780 135673 19 15 66 0 3.20 45.7 13:03:52
4 0 0 21195014 34149832 0 137 0 0 0 0 4884 651401 101881 20 13 67 0 3.04 43.4 13:03:57
6 0 0 21205419 34138651 0 244 0 0 0 0 6644 808653 105196 23 17 60 0 3.62 51.7 13:04:02
4 0 0 21200231 34143314 0 575 0 0 0 0 4102 962335 109243 25 16 59 0 3.70 52.9 13:04:07

14
Usually AIX servers tend to have very small tuning requirements, as most of
workload are Enterprise related. Therefore it must follow strict settings in order to
keep supportability.
However, under most scenarios the following changes are performed
Usual Changes into default configuration
lsdev | awk '
{
if ( $0 ~ /Available/ )
{
if ( $0 ~ /ênt/ )
{
if ( $0 ~ /l-lan/ ) {
system("chdev -P -l "$1" -a max_buf_small=4096");
system("chdev -P -l "$1" -a max_buf_tiny=4096");
system("chdev -P -l "$1" -a min_buf_small=4096");
system("chdev -P -l "$1" -a min_buf_tiny=4096");
}
} else if (( $0 ~ /ên/ )&&( $0 !~ /ênt/ )) {
system("chdev -l "$1" -a mtu_bypass=on");
system("chdev -l "$1" -a rfc1323=1");
system("chdev -l "$1" -a tcp_recvspace=5242880");
system("chdev -l "$1" -a tcp_sendspace=524288");
system("chdev -l "$1" -a tcp_nodelay=0");
system("chdev -l "$1" -a remmtu=1500");
system("chdev -l "$1" -a thread=on");
} else if ( $1 ~ /^vscsi/ ) {
system("chdev -Pl "$1" -a vscsi_err_recov=fast_fail -a vscsi_path_to=60");
}
}
}'
no -p -o rfc1323=1
no -p -o tcp_recvspace=5242880
no -p -o tcp_sendspace=524288
no -p -o udp_recvspace=10526720
no -p -o udp_sendspace=1052672
no -p -o tcp_ephemeral_low=9000
no -p -o tcp_ephemeral_high=65500
no -p -o udp_ephemeral_low=9000
no -p -o udp_ephemeral_high=65500
no -p -o clean_partial_conns=1
ioo -p -o aio_maxreqs=1048576
ioo -p -o aio_server_inactivity=86400
ioo -p -o posix_aio_server_inactivity=86400
schedo -p -o vpm_throughput_mode=4
schedo -p -o smt_snooze_delay=-1
schedo -p -o vpm_xvcpus=-1
smtctl -t 4 -w boot

23
• http://www.ibm.com/training/events
Session reference links

24
Your feedback about this session is very important to us.
Submit a survey at:
ibmtechu.com

25
Continue the conversation
view event
highlights
talk to tech
experts
connect with
attendees
read training
articles
IBM Systems
Technical Events
LinkedIn
community
Join today
bit.ly/IBMTechUconnect

26
ibm.com/training
provides a comprehensive
portfolio of skills and career
accelerators that are designed
to meet all your training needs.
If you can’t find the training that is right for you
with our Global Training Providers, we can help.
Contact IBM Training at dpmc@us.ibm.com
Continue growing your IBM skills

AIX Performance Tuning Session at STU2017

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie AIX Performance Tuning Session at STU2017

Ähnlich wie AIX Performance Tuning Session at STU2017 (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

AIX Performance Tuning Session at STU2017