Surviving the Crisis With the Help of Oracle Database Resource Manager

©2010TietoCorporation
Surviving the Crisis
with the Help of
Oracle Database
Resource Manager
UKOUG Conference Series
Technology & E-Business Suite 2010
Māris Elsiņš
Senior Oracle Applications DBA
Tieto Latvia,
Maris.Elsins@tieto.com

© 2010 Tieto Corporation
Who I am?
• 8 years in IT
• 3 years – PL/SQL developer
• 5 years – Oracle [Apps] DBA (started with 11.5.7 and 8.1.7)
• Certificates
• 10g OCM
• 9i / 10g / 11g OCP
• 11i Applications Database Administrator OCP
• 11i System Administrator OCE
• Conferences
• UKOUG 2007/2008/2010
• LVOUG 2009/2010
• EMEA Harmony 2010
• Current employer – Tieto Latvia
• All kinds of oracle DBA tasks - patching, upgrade, performance
tuning, troubleshooting, planning and implementation of backup and recovery
procedures, cross platform migration, etc.
• Planning of system architecture, design and implementation of HA solutions
(RAC, Data Guard, Custom cold failover)
• Implementation of system specific monitoring, automation of routine tasks
• Technical project planning and coordination, management of team of DBAs
2 2010-11-29

I come from Latvia!
Latvia is a small country…
…with a huge economic crisis.
That‟s where the topic comes from!
Surviving the Crisis with the Help of Oracle
Database Resource Manager
3 2010-11-29

Contents
• Presentation will contain:
• Short Introduction to the Resource Manager
• DEMO
• Evaluation of Performance impact
• The Goal is:
• Not to read the documentation for you…
• …but to show why and when the resource manager can be very
useful
4 2010-11-29

The problem
• OS doesn‟t care enough about DB sessions/processes
according to what business requires
• Assigns the same priority for all processes
• CPU resources are equally distributed among all processes
• Inability to manage DB-specific resources
• Parallel processes
• Active sessions
• Sessions holding latches
• Undo usage
• Context switching overhead when many processes running
• Problems start when there‟s not enough CPU gor everyone
• CPU starvation can be hard to recover from (snowball
effect)
• CPU starvation makes online troubleshooting hard to do
5 2010-11-29

Some possible problem scenarios
• Some samples of problems
• running reports cause too much load on the OLTP system
• One of the sessions allocate all parallel query slaves therefore other
sessions can‟t use any of them
• Application support team runs heavy queries to anayze the data leaving less
resources for business processing
• Too wide search criteria cause “hangs” in the search form
• 3 of 8 CPU cores are idle, my query runs without parallel execution, I could
use the idle CPUs to provide results faster.
• Users don‟t log out and leave idle sessions
• Sessions with incomplete transactions have locked some rows and other
sessions have stuck therefore.
• Does Oracle Resource Manager solve all these problems?
6 2010-11-29
• Some samples of problems
• running reports cause too much load on the OLTP system
• One of the sessions allocate all parallel query slaves therefore other
sessions can‟t use any of them
• Application support team runs heavy queries to anayze the data leaving less
resources for business processing
• Too wide search criteria cause “hangs” in the search form
• 3 of 8 CPU cores are idle, my query runs without parallel execution, I could
use the idle CPUs to provide results faster.
• Users don‟t log out and leave idle sessions
• Sessions with incomplete transactions have locked some rows and other
sessions have stuck therefore.
• Does Oracle Resource Manager solve all these problems?

The concept of the Resource manager
• Resource manager
• Included in Oracle EE licnese
• Allows prioritization of sessions according to the defined business
requirements
• Allows defining the guaranteed amount of allocated resources for
each type of sessions (consumer group)
• Resources not used by higher priority sessions, can be used by
lower priority sessions
• Prioritization archieved by changing the process states to
running/sleeping
• Resource manager does not solve the «lack of CPU
resources» problem, it just controls there execution queue
• Resource manager uses some resources too, the 3rd part of
the presentation will estimate the overhead
7 2010-11-29

Which side is the RM enabled for?
8 2010-11-29
48 active sessions all doing CPU work in both cases
Check Running/Sleeping process states!

The concept of the Resource manager
• Consumer group
• Set of sessions having similar
requirements for server resources
• Resources are allocated to the
consumer group, not individual
sessions
• dba_rsrc_consumer_groups
• Directives
• Rules that define resource allocation
to the consumer group
• dba_rsrc_plan_directives
• Resource plan
• Set of directives defining the
distribution of resources among
consumer groups
• dba_rsrc_plans
9 2010-11-29

Features of Resource Manager
10 2010-11-29
9.2 10.2 11.1 11.2
CPU resource allocation to consumer groups by ratio or
percentage
   
Limit of the degree of parallelism    
active session pool    
Automated change of consumer group if session has
used or is estimated to use the defined amount of
resources (CPU, Est CPU, IO_MB, IO_REQ)
CPU,
Est CPU
CPU,
Est CPU
CPU,
Est CPU,
IO_MB,
IO_REQ
CPU,
Est CPU,
IO_MB,
IO_REQ
Limit of estimated execution time    
Limit size of undo used by uncommitted sessions    
Termination of idle sessions   
Termination of idle blocking sessions   
L0 70% CPU _ORACLE_BACKGROUND_GROUP_ hidden
consumer group for background processes
 
Instance caging /CPU_COUNT + resource plan/ 
Max CPU Utilization limit 

Setting up the resource manager
• Create the resource consumer groups, resource plan and
directives
• Use DB Console or EM Grid Control
• Use API (DBMS_RESOURCE_MANAGER package)
• Define rules to assign sessions to particular consumer
groups
• Connect time
(ORACLE_USER, SERVICE_NAME, CLIENT_OS_USER, CLIENT_PROGRAM, CLIENT_MACHI
NE)
• Run time
(MODULE_NAME, MODULE_NAME_ACTION, SERVICE_MODULE, SERVICE_MODULE_ACTI
ON)
• Enable resource plan
• ALTER SYSTEM SET resource_manager_plan=„ALL_OK';
Excellent documentation!
Oracle® Database Administrator's Guide 11g Release 2 (11.2)
27 Managing Resource Allocation with Oracle Database Resource Manager
http://download.oracle.com/docs/cd/E11882_01/server.112/e17120/dbrm.htm#ADMIN027
11 2010-11-29

Important to keep in mind
• 11g has resource manager enabled by default for
maintenance tasks (can cause problems after upgrade to
11g)
• MAINTENANCE_WINDOW_GROUP scheduler window group
• DEFAULT_MAINTENANCE_PLAN plan
• Waitevent «rsmgr: cpu quantum» for sleeping processes
• Common mistake is to forget about v$session.state
• WAITING – session is currently waiting
• WAITED KNOWN TIME – session is running on CPU
• WAITED SHORT TIME – session is running on CPU
• (same warning is valid for other wait events too)
12 2010-11-29

DEMO
13 2010-11-29

DEMO on SAMPLE_PLAN3
• SYS_GROUP
• 5% L1 CPU
• ONLINE3
• 95% L1 CPU
• Switch the session to LONG3
group if the execution takes
longer then 30 seconds
• OTHER_GROUPS
• 100% L2 CPU
• KILL blocking session if it‟s
idle for longer then 90 seconds
• LONG3
• 20% L1 CPU
• KILL session inactive for
longer then then 20 seconds
14 2010-11-29

Evaluation of performance impact
15 2010-11-29

Evaluation of performance impact
• Test platforms
• 11.1.0.7 on RHEL 4.4, 4Cores + HT, RAID5 Local HDD, 6G RAM
• 11.2.0.1 on HP-UX v11.31, vPAR on ia64 hp superdome server
SD32B, 4Cores, NetApp storage (could have affected the test results), 32G
RAM
• Concept
• Identically repeatable workload
• Testing is the only activity (no schedules, jobs, awr snapshots, etc.)
• Each test repeated 3 times, average execution time recorded
• DBs configured with ASYNC and DIRECT IO (FS cache does not
affect testing)
• DB Buffer cache flushed before every IO test
• Goal
• Identify if enabling the resource manager causes visible impact on
performance
16 2010-11-29

Tests
• «CPU bottleneck» on RHEL with 8, 24, 48 processes
• No RM plan in use
• Simple RM plan in use, processes evenly distributed among CGs
• Complex RM plan in use, processes evenly distributed among CGs
• «CPU bottleneck» on HP-UX with 4, 12, 24, 48 processes
• Simple RM plan in use, processes evenly distributed among CGs
• Complex RM plan in use, processes evenly distributed among CGs
• «IO bottleneck» on RHEL with 24 processes
• Complex RM plan in use
• «IO bottleneck» on HP-UX with 24 processes
• Complex RM plan in use
17 2010-11-29

Testing with simple RM plan
18 2010-11-29

Testing with complex RM plan
19 2010-11-29

«CPU bottleneck»
---noramloadcpu.sql---
set verify off feed off timing on serverout on time on
declare
n number;
x number:=10000000*&1;
z number;
date1 date;
begin
select sysdate into date1 from dual;
DBMS_APPLICATION_INFO.SET_MODULE(module_name => 'TEST',
action_name => '&2');
for n in x+1..x+10000000 loop
z:=sqrt(n);
end loop;
dbms_output.put_line(to_char(date1,'HH24:MI:SS')||' : &&2 :
&&1');
end;
/
sqlplus -s apps/nopassword@TTBI @noramloadcpu.sql 1 Online_group &
sqlplus -s apps/nopassword@TTBI @noramloadcpu.sql 2 Batch_group &
sqlplus -s apps/nopassword@TTBI @noramloadcpu.sql 3 Bug_Maint_group &
sqlplus -s apps/nopassword@TTBI @noramloadcpu.sql 4 Users_group &
sqlplus -s apps/nopassword@TTBI @noramloadcpu.sql 5 Postman_group &
sqlplus -s apps/nopassword@TTBI @noramloadcpu.sql 6 Mail_Maint_group &
echo "---------------------------"
wait
echo "---------------------------"
20 2010-11-29
• Practically no IO
• Each process spinning on
CPU

«CPU bottleneck» results
RHEL 4.4 and 11.1.0.7 HP-UX v11.31 and 11.2.0.1
21 2010-11-29

«CPU bottleneck» results
--------------WITHOUT RESOURCE MANAGER--------------
09:28:05 : LOW : 14 Elapsed: 00:00:15.39
09:28:05 : LOW : 19 Elapsed: 00:00:15.49
09:28:06 : HIGH : 43 Elapsed: 00:00:16.93
09:28:07 : HIGH : 46 Elapsed: 00:00:17.45
09:28:07 : HIGH : 41 Elapsed: 00:00:20.42
09:28:05 : LOW : 13 Elapsed: 00:00:24.82
09:28:15 : HIGH : 39 Elapsed: 00:00:17.40
09:28:09 : HIGH : 35 Elapsed: 00:00:24.06
09:28:22 : HIGH : 38 Elapsed: 00:00:11.19
09:28:19 : HIGH : 34 Elapsed: 00:00:17.72
09:28:30 : LOW : 4 Elapsed: 00:00:11.30
09:28:33 : HIGH : 47 Elapsed: 00:00:10.80
09:28:34 : HIGH : 25 Elapsed: 00:00:10.88
09:28:37 : LOW : 20 Elapsed: 00:00:11.03
09:28:41 : HIGH : 29 Elapsed: 00:00:11.49
09:28:44 : LOW : 12 Elapsed: 00:00:11.66
09:28:45 : LOW : 15 Elapsed: 00:00:13.87
09:28:48 : LOW : 11 Elapsed: 00:00:15.74
09:28:50 : LOW : 6 Elapsed: 00:00:17.68
09:28:55 : LOW : 17 Elapsed: 00:00:16.70
09:28:54 : HIGH : 33 Elapsed: 00:00:19.76
09:28:50 : HIGH : 30 Elapsed: 00:00:23.20
09:28:59 : LOW : 5 Elapsed: 00:00:15.60
09:29:03 : HIGH : 37 Elapsed: 00:00:20.48
09:29:08 : LOW : 23 Elapsed: 00:00:15.43
09:29:02 : HIGH : 28 Elapsed: 00:00:21.59
09:29:06 : LOW : 22 Elapsed: 00:00:21.63
09:29:13 : HIGH : 45 Elapsed: 00:00:16.47
09:29:17 : LOW : 1 Elapsed: 00:00:15.28
09:29:15 : LOW : 18 Elapsed: 00:00:18.10
09:29:20 : HIGH : 40 Elapsed: 00:00:16.72
09:29:24 : LOW : 24 Elapsed: 00:00:12.69
09:29:27 : HIGH : 44 Elapsed: 00:00:14.92
09:29:33 : HIGH : 31 Elapsed: 00:00:11.95
09:29:37 : LOW : 8 Elapsed: 00:00:11.04
09:29:36 : LOW : 21 Elapsed: 00:00:14.46
09:29:37 : LOW : 16 Elapsed: 00:00:13.79
09:29:43 : LOW : 7 Elapsed: 00:00:11.88
09:29:45 : HIGH : 36 Elapsed: 00:00:10.88
09:29:50 : HIGH : 32 Elapsed: 00:00:13.22
09:29:51 : LOW : 2 Elapsed: 00:00:13.32
09:29:53 : HIGH : 42 Elapsed: 00:00:11.37
09:29:56 : LOW : 9 Elapsed: 00:00:10.12
09:29:57 : HIGH : 26 Elapsed: 00:00:17.04
09:30:03 : LOW : 3 Elapsed: 00:00:11.84
09:30:05 : LOW : 10 Elapsed: 00:00:10.92
09:29:59 : HIGH : 27 Elapsed: 00:00:17.63
09:30:08 : HIGH : 48 Elapsed: 00:00:13.25
---------------------------
real 2m17.292s
--------------WITH RESOURCE MANAGER--------------
09:24:50 : HIGH : 25 Elapsed: 00:00:12.07
09:24:54 : HIGH : 42 Elapsed: 00:00:14.77
09:24:54 : HIGH : 29 Elapsed: 00:00:17.03
09:24:53 : HIGH : 39 Elapsed: 00:00:20.57
09:24:55 : HIGH : 48 Elapsed: 00:00:19.70
09:25:04 : HIGH : 36 Elapsed: 00:00:23.10
09:25:04 : HIGH : 30 Elapsed: 00:00:23.79
09:25:03 : HIGH : 28 Elapsed: 00:00:24.38
09:25:11 : HIGH : 38 Elapsed: 00:00:30.34
09:25:11 : HIGH : 44 Elapsed: 00:00:31.58
09:25:12 : HIGH : 35 Elapsed: 00:00:31.28
09:25:19 : HIGH : 46 Elapsed: 00:00:38.93
09:25:20 : HIGH : 40 Elapsed: 00:00:38.48
09:25:21 : HIGH : 34 Elapsed: 00:00:38.87
09:25:23 : HIGH : 45 Elapsed: 00:00:37.68
09:25:23 : HIGH : 32 Elapsed: 00:00:37.37
09:25:22 : HIGH : 31 Elapsed: 00:00:39.74
09:25:26 : HIGH : 47 Elapsed: 00:00:35.94
09:25:23 : HIGH : 33 Elapsed: 00:00:39.29
09:25:25 : HIGH : 37 Elapsed: 00:00:38.00
09:25:29 : HIGH : 26 Elapsed: 00:00:34.94
09:25:28 : HIGH : 27 Elapsed: 00:00:35.72
09:25:31 : HIGH : 41 Elapsed: 00:00:34.19
09:25:49 : HIGH : 43 Elapsed: 00:00:21.17
09:24:50 : LOW : 24 Elapsed: 00:01:55.38
09:24:51 : LOW : 21 Elapsed: 00:02:01.84
09:24:50 : LOW : 22 Elapsed: 00:02:03.07
09:24:53 : LOW : 8 Elapsed: 00:02:04.94
09:24:53 : LOW : 5 Elapsed: 00:02:04.83
09:25:05 : LOW : 3 Elapsed: 00:01:53.88
09:25:17 : LOW : 2 Elapsed: 00:01:44.03
09:25:28 : LOW : 7 Elapsed: 00:01:34.11
09:25:17 : LOW : 9 Elapsed: 00:01:45.86
09:25:19 : LOW : 6 Elapsed: 00:01:43.90
09:25:19 : LOW : 4 Elapsed: 00:01:44.02
09:25:01 : LOW : 10 Elapsed: 00:02:02.05
09:25:03 : LOW : 1 Elapsed: 00:02:00.22
09:25:22 : LOW : 20 Elapsed: 00:01:41.60
09:24:55 : LOW : 16 Elapsed: 00:02:08.80
09:24:55 : LOW : 11 Elapsed: 00:02:08.01
09:25:29 : LOW : 18 Elapsed: 00:01:34.74
09:25:10 : LOW : 15 Elapsed: 00:01:54.69
09:25:12 : LOW : 17 Elapsed: 00:01:51.99
09:25:09 : LOW : 23 Elapsed: 00:01:55.03
09:25:21 : LOW : 12 Elapsed: 00:01:42.87
09:25:11 : LOW : 19 Elapsed: 00:01:53.68
09:25:25 : LOW : 13 Elapsed: 00:01:39.45
09:25:15 : LOW : 14 Elapsed: 00:01:50.21
---------------------------
real 2m15.818s
22 2010-11-29
First
Last
First
Last

«IO bottleneck»
---prepare data---
create table MEL_IO_TEST as select cast(rownum as
number)x, lpad('a',400,'a') y from dual connect by level
<=2654208;
insert into MEL_IO_TEST select * from MEL_IO_TEST;
commit;
create index MEL_IO_TEST_IDX on MEL_IO_TEST(x);
--- ioloadtest.sql---
set verify off feed off timing on serverout on time on
declare
nn number;
date1 date;
begin
select sysdate into date1 from dual;
DBMS_APPLICATION_INFO.SET_MODULE(module_name => 'TEST',
action_name => '&3');
select /*+ index(a
MEL_IO_TEST_IDX)*/avg(length(y)+length(x)) into nn from
MEL_IO_TEST a where x between 2654208/&2*(&1-1) and
2654208/&&2*&&1;
dbms_output.put_line(to_char(date1,'HH24:MI:SS')||' :
&&3 : &&1 : &&2');
end;
/
sqlplus -s apps/apps@MEL @ioloadtest_flush.sql
sqlplus -s apps/apps@MEL @ioloadtest.sql 1 24 Online_group &
sqlplus -s apps/apps@MEL @ioloadtest.sql 2 24 Batch_group &
sqlplus -s apps/apps@MEL @ioloadtest.sql 3 24 Bug_Maint_group &
sqlplus -s apps/apps@MEL @ioloadtest.sql 4 24 Users_group &
sqlplus -s apps/apps@MEL @ioloadtest.sql 5 24 Postman_group &
sqlplus -s apps/apps@MEL @ioloadtest.sql 6 24 Mail_Maint_group &
echo "------------"
wait
echo "------------"
23 2010-11-29

«IO bottleneck» results
24 2010-11-29
RHEL 4.4 and 11.1.0.7
• Average execution time:
• No RM: 227.09s
• Complex RM: 226.68s
HP-UX v11.31 and 11.2.0.1
• Average execution time:
• No RM: 36,30s
• Complex RM: 37,42s

«IO bottleneck» results
------------WITHOUT RESOURCE MANAGER------------
13:23:47 : HIGH : 23 : 64 Elapsed: 00:02:34.51
13:23:47 : LOW : 1 : 64 Elapsed: 00:02:55.07
13:23:47 : LOW : 2 : 64 Elapsed: 00:02:57.05
13:23:47 : HIGH : 24 : 64 Elapsed: 00:03:27.20
13:23:47 : LOW : 12 : 64 Elapsed: 00:03:43.38
13:23:47 : LOW : 8 : 64 Elapsed: 00:03:44.79
13:23:47 : LOW : 10 : 64 Elapsed: 00:03:44.81
13:23:47 : LOW : 11 : 64 Elapsed: 00:03:45.02
13:23:47 : LOW : 9 : 64 Elapsed: 00:03:45.07
13:23:47 : HIGH : 16 : 64 Elapsed: 00:03:45.23
13:23:47 : HIGH : 14 : 64 Elapsed: 00:03:45.26
13:23:47 : HIGH : 15 : 64 Elapsed: 00:03:45.34
13:23:47 : LOW : 7 : 64 Elapsed: 00:03:45.42
13:23:47 : HIGH : 13 : 64 Elapsed: 00:03:45.61
13:23:47 : HIGH : 20 : 64 Elapsed: 00:03:46.35
13:23:47 : HIGH : 19 : 64 Elapsed: 00:03:46.44
13:23:47 : LOW : 5 : 64 Elapsed: 00:03:46.45
13:23:47 : LOW : 4 : 64 Elapsed: 00:03:46.80
13:23:47 : HIGH : 22 : 64 Elapsed: 00:03:47.05
13:23:47 : HIGH : 18 : 64 Elapsed: 00:03:47.21
13:23:47 : LOW : 6 : 64 Elapsed: 00:03:47.35
13:23:47 : HIGH : 17 : 64 Elapsed: 00:03:47.35
13:23:47 : LOW : 3 : 64 Elapsed: 00:03:47.38
13:23:47 : HIGH : 21 : 64 Elapsed: 00:03:47.54
------------
real 3m48.504s
------------WITH RESOURCE MANAGER------------
13:32:14 : LOW : 7 : 64 Elapsed: 00:02:52.47
13:32:14 : LOW : 2 : 64 Elapsed: 00:02:52.82
13:32:15 : HIGH : 24 : 64 Elapsed: 00:03:23.36
13:32:15 : HIGH : 23 : 64 Elapsed: 00:03:25.97
13:32:14 : HIGH : 13 : 64 Elapsed: 00:03:43.74
13:32:15 : HIGH : 16 : 64 Elapsed: 00:03:43.82
13:32:14 : LOW : 9 : 64 Elapsed: 00:03:44.03
13:32:15 : HIGH : 22 : 64 Elapsed: 00:03:44.09
13:32:14 : HIGH : 19 : 64 Elapsed: 00:03:44.60
13:32:14 : LOW : 10 : 64 Elapsed: 00:03:44.72
13:32:14 : LOW : 11 : 64 Elapsed: 00:03:45.05
13:32:15 : HIGH : 21 : 64 Elapsed: 00:03:45.12
13:32:15 : LOW : 3 : 64 Elapsed: 00:03:45.32
13:32:15 : HIGH : 20 : 64 Elapsed: 00:03:45.37
13:32:14 : LOW : 8 : 64 Elapsed: 00:03:45.83
13:32:14 : LOW : 6 : 64 Elapsed: 00:03:45.89
13:32:15 : HIGH : 17 : 64 Elapsed: 00:03:45.68
13:32:14 : HIGH : 14 : 64 Elapsed: 00:03:45.85
13:32:14 : LOW : 12 : 64 Elapsed: 00:03:46.09
13:32:14 : LOW : 1 : 64 Elapsed: 00:03:46.28
13:32:15 : HIGH : 15 : 64 Elapsed: 00:03:46.12
13:32:15 : HIGH : 18 : 64 Elapsed: 00:03:46.30
13:32:14 : LOW : 4 : 64 Elapsed: 00:03:46.44
13:32:14 : LOW : 5 : 64 Elapsed: 00:03:46.46
------------
real 3m47.104s
25 2010-11-29
• RM does not help if the system is IO-bound

Summary
• Very useful if CPU utilization close to 100%
• Ensures most critical sessions get the resources
• Reduces overall load on the server by setting least important
processes to «sleep»
• Resource Manager does not solve the problem – just allows
prioritization
• Performance overhead is practically invisible, (could depend
on platform) – test yourself!
• Included in EE license
• No downtime needed to implement it, setup can be changed
dynamically
• Can be used as a short-term workaround of some
performance problems.
• Some bugs can show up as additional code layer introduced
– test it!
26 2010-11-29

Surviving the Crisis With the Help of Oracle Database Resource Manager

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie Surviving the Crisis With the Help of Oracle Database Resource Manager

Ähnlich wie Surviving the Crisis With the Help of Oracle Database Resource Manager (20)

Mehr von Maris Elsins

Mehr von Maris Elsins (15)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Surviving the Crisis With the Help of Oracle Database Resource Manager