Proof of Concept with Real Application Testing 12c

PROOF OF CONCEPT
with Real Application Testing 12c
Luís Marques - @drune - http://lcmarques.com

Me
Name: Luís Marques
http://lcmarques.com / @drune / lcarapinha@gmail.com

We will discuss
Database Replay features on 12c version and how we
can apply stress test and workload scale-up techniques to
ensure a real world proof of concept
SQL Perfomance Analyzer
Actual PoC results
We will not to discuss

PoC – What really is?
An huge amount of work to plan and execute (and re-
execute until the end of available time period)
Most work is for DBA if Real Application Testing is used.

The (real) way it was…
Hardware arrives
Setup required
Recent Backup
required
Restore your DB into
new hardware
Flashback Guaranteed
Restore PointSetup Application:
- New hardware
- Install and Configure:
- AS, Clients, Network, etc
We are
ready!
2 weeks of infinite fun
and infinite problems

The (real) way it was…
Database scheduler Jobs
created on-the-fly
Small reference
in docs
(not even in
restrictions area)

The way it should be
Working with the vendor to provide support for Real
Application Testing
Client requested a simple and effective way to test new
platform and software releases

The way it should be
Hardware arrives
Setup required
Recent Backup
required
Restore your DB into
new hardware using
START_SCN from Capture
Flashback Guaranteed
Restore Point
DB Replay
DB Capture

Database Replay - Capture

Before start any capture…
RAT Patches on Production and PoC database (Doc ID
560977.1)
Capture Restrictions: direct path load external files with
SQL*Loader, scheduler jobs, flashback querys, distributed trx, etc..
Workload is important: sysdate use,avoid maintenance windows
or EM activity
No ongoing transactions during capture - Plan you
database shutdown and start capture immediately after startup

Workload Capture
Payment processing starts at 19:00:
18:35 – SQL> shutdown immediate
18:45 – SQL> startup restrict
18:50 – $ sqlplus / as sysdba
18:50 – SQL> @capture_payment_day.sql
DBMS_WORKLOAD_CAPTURE.ADD_FILTER ('PaymentUser',
'USER’, 'PAY_A');
DBMS_WORKLOAD_CAPTURE.START_CAPTURE
(name =>'capture_payment_day',
dir => 'DB_REPLAY_DIR',
auto_unrestrict => TRUE,
default_action => 'EXCLUDE',
sts_cap_interval => 300);

Workload Capture
Populate DBA_WORKLOAD_CAPTURES / FILTERS
SQL> exec dbms_workload_capture.get_capture_info('DB_REPLAY_DIR');
Save the START_SCN to use with RMAN UNTIL SCN
to restore your database on PoC database.

Workload Capture
Export AWR data to compare performance between
Production and PoC database:
SQL> exec dbms_workload_capture.export_awr(2);
Dump of AWR will be used after replay to compare AWR on
PoC database
AWR data

Workload Capture – The intelligent stuff
Workload Intelligence – Java programs to analyze
Captured data
Don’t run it on production system (Heavy memory
consumption)
Allow you to know more about the capture by patterns
Create user
wi
Create
Workload
Job
Create
Workload
Model
Identify
Patterns
Generate
Report
Java –classpath … oracle.dbreplay.workload.intelligence.LoadInfo -job wi_job1
-cdir /home/oracle/scripts/rat/db_replay_dir -cstr
jdbc:oracle:thin:@baco:1521:bacodb2 -user wi
Java –classpath … oracle.dbreplay.workload.intelligence.BuildModel -job
wi_job1 -cstr jdbc:oracle:thin:@baco:1521:bacodb2 -user wi
Java –classpath … oracle.dbreplay.workload.intelligence.FindPatterns -job
wi_job1 -cstr jdbc:oracle:thin:@baco:1521:bacodb2 -user wi
Java –classpath … oracle.dbreplay.workload.intelligence.GenerateReport -job
wi_job1 -cstr jdbc:oracle:thin:@baco:1521:bacodb2 -user wi –top 5 –out
intel_report.html

How an intelligence report looks…
Pattern is a sequence of transactions that run under “certain order”.
Allow you to identify mostly used “code paths” – Example: create a customer or
process a visa card “code paths”
Patterns are ordered by different columns (DB Time, Nr. Executions and Length)
Pattern id = 2  Create a customer run 2333 times during the capture with a DB
Time 9 seconds (0,78% of total DB Time)
Pattern id = 2
Pattern id =1

Database Replay – Replay

Before start any replay…
Setup guarantee restore points before any replay
Run java utility “Workload Analyzer” – Check the
quality
of capture (insufficient data, errors, etc)
Run a small test replay (~ 20 min) before full replay -
Avoid potential issues with full replay
Set up your system date to the same time as time of
capture start if application relies on sysdate
Before performance analysis perform a replay analysis
and review divergences

Before start replay - sysdate
Set system clock on PoC system to the start of time of
capture
Make sure that NTP daemon is disabled
Make sure that AWR snapshots are still created
automatically after system clock changed
Restart PoC database after change in the clock.

Before start replay – The Workload
Analyzer
Java utility provided by Oracle to analyze the quality of
your capture.
$ java -classpath
$ORACLE_HOME/jdbc/lib/ojdbc6.jar:$ORACLE_HOME/rdbms/jlib/dbrparser.jar:$ORACLE_H
OME/rdbms/jlib/dbranalyzer.jar: oracle.dbreplay.workload.checker.CaptureChecker
/home/oracle/scripts/rat/db_replay_dir
jdbc:oracle:thin:@baco.localdomain:1521:prodpay2
Almost all sessions have been captured in-flight.
PL/SQL blocks or functions have 'complicated' logic or
multiple commits in them, they are hard to synchronize and
they behavior might change during replay.
wcr_cap_analysis.html

Captured workload: Sequential payment processing by Card type
Goal: Parallelized payment processing by Card Type
Scenario #1

Consolidate Replay – Workload folding
Consolidate Database Replay: Slicing an existing captured
workload into subsets by specifying a point in time
Allows scale-in testing replaying workload subsets together
Our Goal: Parallelized payment processing by Card Type
Start of Day
Processing
VISA Payment
Process
MasterCard
Payment
Process
OthersCard
Payment
Process
Workload
TimeWL Subset
#1
WL Subset
#2
WL Subset
#3

Workload from 1 day (7pm to 6am) – capture_payment_day
3 subsets of workload derived from 1 capture:
Start of Day
Processing
VISA Payment
Process
MasterCard
Payment
Process
OthersCard
Payment
Process
Workload
Time
Day Finish
Replay this!

BEGIN_TIME and END_TIME are offset in seconds from
the start of the capture (tricky)
Incomplete calls are discarded to avoid problematic results
Example (VISA workload subset #1):
• 4500 seconds after start of capture (1 hour and 15 minutes)
• Ends after 8100 seconds from the start of capture (1 hour of duration)
SQL> EXEC DBMS_WORKLOAD_REPLAY.GENERATE_CAPTURE_SUBSET (
INPUT_CAPTURE_DIR => 'DB_REPLAY_DIR',
OUTPUT_CAPTURE_DIR => 'DB_REPLAY_VISA',
NEW_CAPTURE_NAME => 'CardVisa_wkld’,
BEGIN_TIME => 4500,
BEGIN_INCLUDE_INCOMPLETE=> FALSE,
END_TIME => 8100,
END_INCLUDE_INCOMPLETE=> FALSE,
PARALLEL_LEVEL=>1)

Preprocess capture from all directories:
EXEC DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE ('DB_REPLAY_VISA');
EXEC DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE ('DB_REPLAY_MCARD');
EXEC DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE ('DB_REPLAY_OTHERS');
Create a consolidate replay schedule
EXEC DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE
('visa_mcard_others_schedule');
SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_VISA') FROM dual;
SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_MCARD') FROM dual;
SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_OTHERS') FROM dual;
EXEC DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE;

Remap connections (DBA_WORKLOAD_CONNECTION_MAP) – You know how to
do it 
Prepare your consolidation replay:
SQL> EXEC
DBMS_WORKLOAD_REPLAY.PREPARE_CONSOLIDATED_REPLAY
(synchronization=>’OBJECT_ID’, think_time_auto_correct => TRUE )
• Each capture subset have a different SCN. SCN-based sync is not possible
• Only OBJECT_ID and OFF are supported as syncronization options
• Connect_time_scale and think_time_scale were left at default
• Best Results avoiding connect time modifications or think time modifications

Consolidate Replay – OBJECT_ID
A “more advanced” synchronization scheme is used
Ensures that SCN’s for different captures do not clash
Better dealing with concurrency when actions do not touch
the same object
Bugs happen:
If you find problems with OBJECT_ID sync you are left to no
syncronization
Monitor the progress of your replay. A collection of scripts
here help - Scripts to Debug Slow Replay (Doc ID 760402.1)

Start replay clients – At least one per subset
wrc system/oracle mode=replay
replaydir=/home/oracle/scripts/rat/db_replay_dir/
sleep 1
wrc system/oracle mode=replay replaydir=/home/oracle/scripts/
rat/db_replay_dir/
…
Start database replay
SQL> EXEC
DBMS_WORKLOAD_REPLAY.START_CONSOLIDATED_REPLAY;

Review divergences after replay
Application Level divergences:
• Example: Check if all the VISA cards were processed (manual SQL Script)
Data Level divergences:
• Different number of rows returned by DML or SQL queries – Don’t match with capture
Error Level divergences:
• Errors on Capture that are not on replay
• Errors on Replay that were not captured
• Different errors on capture and replay
select DBMS_WORKLOAD_REPLAY.REPORT(replay_id => 1, format =>
'HTML') from dual;

Sequential vs Parallel: Compare it!
Import AWR from previous sequential replay to compare to parallel replay
dbms_workload_replay.import_awr(replay_id => 12, staging_schema => 'PAY_A');
Generate a compare replays report
spool report_compare_consolidate.html
VAR v_clob CLOB
BEGIN dbms_workload_replay.compare_period_report (replay_id1 => 11,
replay_id2 => 12,
format => DBMS_WORKLOAD_REPLAY.TYPE_HTML, result => :v_clob );
END;
/
PRINT v_clob
spool off

Sequential vs Parallel: Compare it!
Compare DB Time between periods (Sequential TP1 vs
Parallel TP2);
Easier than read an AWR 

Workload: Parallelized payment processing by Card Type
Goal: Parallelized payment processing by Card Type
and ATM machines payment processing from
another database capture
Scenario #2

Consolidate Replay – Time shifting
Allows shifting a replay to a time in the future and align
with other replays
Stress testing by align peak loads
Peak number of sessions
Add another
workload here!

Align “Multibanco ATM processing” and Card Payment
Processing to start exactly at same time
VISA Payment
Process
MasterCard
Payment
Process
OthersCard
Payment
Process
Validations
Multibanco
ATM
processing
Workload
time
Workload
time
Delay time

Preprocess and delay the workloads to match ATM
replay
EXEC
DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE('payment_peak_align');
SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_ATM') FROM
dual;
SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_VISA', 600)
FROM dual;
SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_MCARD', 650)
FROM dual;
SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_OTHERS', 660)
FROM dual;
EXEC DBMS_WORKLOAD_REPLAY.INITIALIZE_CONSOLIDATED_REPLAY
('payment_peak_replay', 'payment_peak_align');
Delay
secs

Workload: Single workload capture on ATM machines payment
Goal: Stress test ATM machine payment processing on a
very busy day – December 24th
Scenario #3

Consolidate Replay – Schema
Remapping
A single workload consolidated into two different instances of
the application
Original capture schema: MULTIBANCO_ATM (original)
Remapped schema: MULTIBANCO_ATM_B (remapped)
Validations
Multibanco ATM
processing
Multibanco ATM
processing
Validations
Fees Calc
Fees Calc
MULTIBANCO_ATM_B
Workload
time

Consolidate Replay – Schema
Remapping
Add the same workload twice to a consolidate replay
SQL> EXEC
DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE(’multibanco
_stress_schedule');
SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE
('DB_REPLAY_ATM') FROM dual;
SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE
('DB_REPLAY_ATM') FROM dual;
EXEC
DBMS_WORKLOAD_REPLAY.INITIALIZE_CONSOLIDATED_REPLAY
('payment_peak_replay', ‘multibanco_stress_schedule');
EXEC DBMS_WORKLOAD_REPLAY.SET_USER_MAPPING (2,
’MULTIBANCO_ATM', ’MULTIBANCO_ATM_B');

Workload: Applied to all previous scenarios
Goal: Get a “even” more realist parallelized payment
processing
Scenario #4

Query Only – Warm up my buffer cache
When you start your replay, remember that your
cache is "cold”
Replay may not be realistic or take too much time
– capture is done with already some blocks in buffer
cache.
Warm-up the buffer cache (if needed) with Query
Only Replay before the real workload.
Query Only Replay replays read-only statements

Query Only – Warm up my buffer cache
example
EXEC
DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE
('visa_mcard_others_schedule');
SELECT
DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_
VISA’, query_only => TRUE) FROM dual;
SELECT
MCARD’, query_only => TRUE) FROM dual;
SELECT
OTHERS’,query_only => TRUE) FROM dual;
EXEC
DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE;

Final Notes
Consolidate Replay can be really helpful on stress
testing and scaling up on “brand new” PoC system
Comparing periods is important: Capture vs Replay;
Consolidate Replay vs Sequential Replays, etc
Follow capture and replay recommendations. Make
sure that you know your capture and his subsets and start
your replay with the “defaults”
Try different synchronization options if you find in
trouble with OBJECT_ID. Remember that OBJECT_ID
and OFF are the only options for Consolidated Replay

Q & A

Proof of Concept with Real Application Testing 12c

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Andere mochten auch

Andere mochten auch (20)

Ähnlich wie Proof of Concept with Real Application Testing 12c

Ähnlich wie Proof of Concept with Real Application Testing 12c (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Proof of Concept with Real Application Testing 12c

Hinweis der Redaktion