Oracle 11gR2 utilities

1. Road less tRAveled Oracle 11g R2 utilities Guenadi Nedkov Jilevski E-mail: gjilevski@gmail.com BGOUG : Bulgaria April 2010

2. Agenda: Road less traveled Oracle utilities Top new features in Oracle 11g R1 & 11g R2 Monitoring & Performance Tools Autotrace, SQL trace, TKPROF, DBMS_STATS, DBMS_SQLTUNE Oradebug Hidden Oracle 11g R2 parameters Useful V$ and X$ tables for Oracle 11g R2 Powerful system tools: dbx, strace, orakill ORA-0600 Tips and tricks KFOD for ASM Monitoring

3. Top features for Oracle 11g R2 ADR for monitoring Data Recovery Advisor with RMAN for Oracle Backup and Recovery with Oracle 11g Real Application Testing (RAT) features with 11g New partitioning features for Oracle 11g Data partitioning such as (automatic interval, partitioning by reference, composite range-range, composite list-range, composite list-list, composite list-hash, virtual column) and partition advisor Enhancements for Oracle 11g performance tuning (SQLPA,HM, SQL plan management, Server/Client result cache, Enhanced statistics, SQL tuning automation, PL/SQL native compilation etc...) Invisible indexes new feature for Oracle 11g In-Memory Parallel Execution in 11g R2 Auto Degree of Parallelism (DOP) and Queuing in 11g R2 The DBMS_PARALLEL_EXECUTE package in 11g R2 Performance improvement of On-Commit Fast Refresh in 11g R2

4. What is ADR for Oracle 11gR2? ADR is the new Automatic Diagnostic monitor with 11g release. Many new monitoring and reporting features. Completely new paradigm for Oracle DBA monitoring. Available in both command line interface (adrci) or from Oracle 11g Grid/Database control. Replaces alert.log, trace files text based monitoring in releases prior to 11g. Alert.log still available but recommended you learn ADR.

6. Inside Automatic Disgnostic Monitor (ADR) for Oracle 11g R2 ADR stores the main configuration details in log.xml file Default ADR directory set Oracle 11g initialization diagnostic_dest parameter SQL> show parameter diagnostic_dest NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ diagnostic_dest string /u01/app/oracle SQL> SQL> SELECT * FROM V$DIAG_INFO; INST_ID NAME VALUE ---------- ---------------------------------------------------------------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1 Diag Enabled TRUE 1 ADR Base /u01/app/oracle 1 ADR Home /u01/app/oracle/diag/rdbms/racdb/RACDB_1 1 Diag Trace /u01/app/oracle/diag/rdbms/racdb/RACDB_1/trace 1 Diag Alert /u01/app/oracle/diag/rdbms/racdb/RACDB_1/alert 1 Diag Incident /u01/app/oracle/diag/rdbms/racdb/RACDB_1/incident 1 Diag Cdump /u01/app/oracle/diag/rdbms/racdb/RACDB_1/cdump 1 Health Monitor /u01/app/oracle/diag/rdbms/racdb/RACDB_1/hm 1 Default Trace File /u01/app/oracle/diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_6723.trc 1 Active Problem Count 0 1 Active Incident Count 0 11 rows selected.

7. Inside the (ADR) for Oracle 11g R2 ADR home directory located under default: <ADR_BASE>/diag/rdbms/<db_name>/<instance_id> [oracle@raclinux1 RACDB_1]$ pwd /u01/app/oracle/diag/rdbms/racdb/RACDB_1 [oracle@raclinux1 RACDB_1]$ ls alert cdump hm incident incpkg ir lck metadata stage sweep trace [oracle@raclinux1 RACDB_1]$ [oracle@raclinux1 RACDB_1]$ adrci ADRCI: Release 11.2.0.1.0 - Production on Mon Jan 25 15:51:01 2010 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. ADR base = "/u01/app/oracle" adrci>

8. Using the Automatic Diagnostic Repository (ADR) Command line utility (adrci) Help command with ADR: help and extended help adrci> help HELP [topic] Available Topics: CREATE REPORT ECHO EXIT HELP HOST IPS PURGE RUN SET BASE SET BROWSER SET CONTROL SET ECHO SET EDITOR SET HOMES | HOME | HOMEPATH SET TERMOUT SHOW ALERT SHOW BASE SHOW CONTROL SHOW HM_RUN SHOW HOMES | HOME | HOMEPATH SHOW INCDIR SHOW INCIDENT SHOW PROBLEM SHOW REPORT SHOW TRACEFILE SPOOL There are other commands intended to be used directly by Oracle, type "HELP EXTENDED" to see the list adrci>

9. Using extended help with Oracle 11g R2 ADR [oracle@raclinux1 ~]$ adrci adrci> help extended HELP [topic] Available Topics: BEGIN BACKUP CD CREATE STAGING XMLSCHEMA CREATE VIEW DDE DEFINE DELETE DESCRIBE DROP VIEW END BACKUP INSERT LIST DEFINE MERGE ALERT MERGE FILE MIGRATE QUERY REPAIR SELECT SET COLUMN SHOW CATALOG SHOW DUMP SHOW SECTION SHOW TRACE SHOW TRACEMAP SWEEP UNDEFINE UPDATE VIEW adrci>

10. Viewing alert.log and trace files with ADR for Oracle 11g R2 Using ADR to view alert.log Use the following command with ADR: show alert [oracle@raclinux1 ~]$ adrci ADRCI: Release 11.2.0.1.0 - Production on Fri Jan 29 18:10:28 2010 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. ADR base = "/u01/app/oracle" adrci> show alert Choose the alert log from the following homes to view: 1: diag/asm/+asm/+ASM1 2: diag/tnslsnr/raclinux1/listener_scan1 3: diag/tnslsnr/raclinux1/listener 4: diag/rdbms/rac11g/RAC11G 5: diag/rdbms/rac11gr1/RAC11gR1 6: diag/rdbms/rac11gr1/RAC11GR1 7: diag/rdbms/racdb/RACDB_1 Q: to quit Please select option: 7

11. Viewing the Oracle 11g R2 alert.log with ADR 2010-01-24 18:18:32.252000 +02:00 Starting ORACLE instance (normal) 2010-01-24 18:18:33.725000 +02:00 LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Interface type 1 eth1 10.10.2.0 configured from GPnP Profile for use as a cluster interconnect Interface type 1 eth0 192.168.2.0 configured from GPnP Profile for use as a public interface Shared memory segment for instance monitoring created Picked latch-free SCN scheme 2 Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST Autotune of undo retention is turned on. IMODE=BR ILAT =27 LICENSE_MAX_USERS = 0 SYS auditing is disabled 2010-01-24 18:19:07.433000 +02:00 NOTE: Volume support enabled Starting up: Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production With the Partitioning, Real Application Clusters, Oracle Label Security, OLAP, Data Mining, Oracle Database Vault and Real Application Testing options. Using parameter settings in client-side pfile /u01/app/oracle/admin/RACDB/pfile/init.ora on machine raclinux1.gj.com System parameters with non-default values: processes = 150 memory_target = 2336M control_files = "+DATA/racdb/control01.ctl" control_files = "+DATA/racdb/control02.ctl" db_block_size = 8192 compatible = "11.2.0.0.0" log_archive_format = "%t_%s_%r.dbf" db_recovery_file_dest = "+FLASH" db_recovery_file_dest_size= 8000M remote_login_passwordfile= "EXCLUSIVE" db_domain = "" dispatchers = "(PROTOCOL=TCP) (SERVICE=RACDBXDB)" remote_listener = "raclinu-cluster:1521" audit_file_dest = "/u01/app/oracle/admin/RACDB/adump" audit_trail = "DB" db_name = "RACDB" open_cursors = 300 diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 10.10.2.11 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2

12. Viewing trace files with ADR for Oracle 11g R2 Use the show tracefile command for ADR with Oracle 11g adrci> show tracefile diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_4592.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_10352.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_11557.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_5887.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_2678.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_22084.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_16417.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ping_30558.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_25283.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_4914.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_25584.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_qmnc_7024.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_15058.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_25409.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_18403.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_23833.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_23604.trc diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_30828.trc adrci>

13. Incident Reporting with ADR for Oracle 11g R2 ADR uses incidents for critical errors with Oracle 11g R2 These incidents can be packaged by ADR and sent to Oracle support for analysis Useful new feature with Oracle 11g for problem resolution and reporting Perform many of the tasks used by the RDA (Remote Diagnostic Assistant) for releases prior to Oracle 11g.

14. Show incident with ADR and Oracle 11g R2 Critical error appear as incidents. You can use show incident command for ADR with Oracle 11g adrci> set homepath diag/rdbms/racdb/RACDB_1 adrci> show incident ADR Home = /u01/app/oracle/diag/rdbms/racdb/RACDB_1: ************************************************************************* INCIDENT_ID PROBLEM_KEY CREATE_TIME ----------------------------------------------------------------------------------------------------------------------- 11009 ORA 600 [kssadpm: null parent] 2010-01-15 19:08:30.464549 -04:00 1 rows fetched

15. Show incident detail for ADR and Oracle 11g R2 Detail view for incidents with ADR and Oracle 11g adrci> show incident –mode detail –p “incident_id=11009” ADR Home = /u01/app/oracle/diag/rdbms/ora11g/ORA11G: ************************************************************************* ********************************************************** INCIDENT INFO RECORD 1 ********************************************************** INCIDENT_ID 11009 STATUS ready CREATE_TIME 2010-01-15 19:08:30.464549 -04:00 PROBLEM_ID 1

16. Packaging Incidents for Oracle Support ADR provide the following commands for managing incidents: IPS CREATE PACKAGE for new incident with Oracle 11g IPS ADD INCIDENT <insident_id> package <package_id> IPS GENERATE PACKAGE <package_id> in <directory> This procedure creates a ZIP file with the trace diagnostics for Oracle support to analyze for critical errors. Use the IPS SHOW CONFIGURATION command to check configuration for ADR incidents.

17. Example: Incident packages for ADR and Oracle 11g R2 First we need to create a new package for the incident: adrci> ips create package Created package 1 without any contents, correlation level typical Then we need to add the incident to the new package for ADR: adrci> ips add incident 11009 package 1; Added incident 11009 to package 1 Finally, we need to generate the package with data: adrci> ips generate package 1 in /u01/app/oracle/adrci/support Generated package 1 in file /u01/app/oracle/adrci/support/IPSPKG_20080817214731_COM_1.zip, mode complete The new file is generated as a zip file in the directory for support

18. Performance toolkit for Oracle 11g R2 Every DBA should use these tools for tuning Oracle! Autotrace SQL trace TKPROF DBMS_STATS DBMS_SQLTUNE package ORADEBUG

19. Using Autotrace with Oracle 11g R2 Autotrace useful quick and dirty tool for tracing with Oracle 11g Provides many tracing and tuning options: Explain plan Costs associated with SQL queries Tuning statistics Used with SQL*Plus interface Excellent starting point for tuning SQL queries.

20. Using Autotrace with Oracle 11g R2 Use the set autotrace on command in SQL*Plus to enable Autotrace and set autotrace off to disable autotrace tool. Syntax for autotrace: SQL> set autotrace Usage: SET AUTOT[RACE] {OFF | ON | TRACE[ONLY]} [EXP[LAIN]] [STAT[ISTICS]] SQL>

21. Autotrace Example for Oracle 11g R2 SQL> set autotrace on SQL> select prod_name, prod_list_price from sh.products where prod_list_price > 7.99 and prod_list_price < 69.99;

22. Autotrace with Oracle 11g R2 52 rows selected. Execution Plan ---------------------------------------------------------- Plan hash value: 1954719464 ------------------------------------------------------------------------------ | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | ------------------------------------------------------------------------------ | 0 | SELECT STATEMENT | | 3 | 93 | 3 (0)| 00:00:01 | |* 1 | TABLE ACCESS FULL| PRODUCTS | 3 | 93 | 3 (0)| 00:00:01 | ------------------------------------------------------------------------------ Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter("PROD_LIST_PRICE"<69.99 AND "PROD_LIST_PRICE">7.99) Statistics ---------------------------------------------------------- 7880 recursive calls 0 db block gets 1769 consistent gets 63 physical reads 0 redo size 2555 bytes sent via SQL*Net to client 452 bytes received via SQL*Net from client 5 SQL*Net roundtrips to/from client 88 sorts (memory) 0 sorts (disk) 52 rows processed SQL>

23. How to use SQL trace with Oracle 11g R2 SQL Trace is powerful tool for the serious Oracle Performance expert Event based: 10046 (SQL Trace), 10053 ( Cost based Optimizer – CBO) Can be set at instance or session level Use DBMS_SUPPORT, DBMS_SYSTEM, DBMS_SESSION, ORADEBUG, or SQL*PLUS commands to enable tracing activities with Oracle 11g R2.

24. PL/SQL packages for tracing in 11gR2 SQL> EXEC DBMS_SESSION.set_sql_trace(sql_trace => TRUE); SQL> EXEC DBMS_SESSION.set_sql_trace(sql_trace => FALSE); SQL> EXEC DBMS_SYSTEM.set_sql_trace_in_session(sid=>123, serial#=>1234, sql_trace=>TRUE); SQL> EXEC DBMS_SYSTEM.set_sql_trace_in_session(sid=>123, serial#=>1234, sql_trace=>FALSE); SQL> EXEC DBMS_SYSTEM.set_ev(si=>123, se=>1234, ev=>10046, le=>8, nm=>' '); SQL> EXEC DBMS_SYSTEM.set_ev(si=>123, se=>1234, ev=>10046, le=>0, nm=>' '); -- All versions, requires DBMS_SUPPORT package to be loaded. SQL> EXEC DBMS_SUPPORT.start_trace(waits=>TRUE, binds=>FALSE); SQL> EXEC DBMS_SUPPORT.stop_trace; SQL> EXEC DBMS_SUPPORT.start_trace_in_session(sid=>123, serial=>1234, waits=>TRUE, binds=>FALSE); SQL> EXEC DBMS_SUPPORT.stop_trace_in_session(sid=>123, serial=>1234); The DBMS_SUPPORT package is not present by default, but can be loaded as the SYS user by executing the @$ORACLE_HOME/rdbms/admin/dbmssupp.sql script. For methods that require tracing levels the following are valid values: 0 - No trace. Like switching sql_trace off. 2 - The equivalent of regular sql_trace. 4 - The same as 2, but with the addition of bind variable values. 8 - The same as 2, but with the addition of wait events. 12 - The same as 2, but with both bind variable values and wait events.

25. How to perform SQL trace for Oracle 11g R2 Oracle Metalink Note 199081.1 provides good overview for SQL trace Using SQL*PLUS commands for SQL trace at session/instance levels: alter system/session set sql_trace=true; alter system/session set timed_statistcs=true; Optional for SQL timing tracing details ALTER SYSTEM/SESSION SET EVENTS’10046 trace name context forever, level <level>’; Output SQL trace files to user_dump_dest as .trc file SHOW PARAMETER USER_DUMP_DEST

26. SQL trace example for Oracle 11g R2 Should use SYS enabled account to run the SQL trace. Need to have explain plan enabled (i.e.: utlxplan.sql) Enable parameters for SQL tracing: timed_statistics, sql_trace Recommended use ALTER SESSION as ALTER SYSTEM can place load on Oracle 11g system and impact performance SQL> alter session set timed_statistics=true; Session altered. SQL> alter session set sql_trace=true; Session altered. SQL> show parameter user_dump_dest NAME TYPE VALUE ----------------------------------------------------------------------------- user_dump_dest string /u01/app/oracle/diag/rdbms/ora 11g/ORA11G/trace

27. Contents of SQL trace file Use the ALTER SESSION SET TRACEFILE_IDENTIFIER command to name trace file for easy identification SQL> alter session set tracefile_identifire=‘products_sqltune’; Session altered SQL> show parameter tracefile_identifier content of sample SQL Trace file ===================== PARSING IN CURSOR #2 len=109 dep=0 uid=88 oct=3 lid=88 tim=1264443780269154 hv=1524029202 ad='8e65be74' sqlid='gkc6r0xdddpsk' select prod_name, prod_list_price from sh.products where prod_list_price > 7.99 and prod_list_price < 69.99 END OF STMT PARSE #2:c=0,e=42,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1954719464,tim=1264443780269152 EXEC #2:c=0,e=24,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=1954719464,tim=1264443780269240 FETCH #2:c=0,e=45,p=0,cr=3,cu=0,mis=0,r=1,dep=0,og=1,plh=1954719464,tim=1264443780269346 FETCH #2:c=0,e=19,p=0,cr=1,cu=0,mis=0,r=15,dep=0,og=1,plh=1954719464,tim=1264443780269523 FETCH #2:c=0,e=26,p=0,cr=2,cu=0,mis=0,r=15,dep=0,og=1,plh=1954719464,tim=1264443780269889 FETCH #2:c=0,e=16,p=0,cr=1,cu=0,mis=0,r=15,dep=0,og=1,plh=1954719464,tim=1264443780270211 FETCH #2:c=0,e=14,p=0,cr=1,cu=0,mis=0,r=6,dep=0,og=1,plh=1954719464,tim=1264443780270463 *** 2010-01-25 20:23:01.766 CLOSE #2:c=0,e=13,dep=0,type=3,tim=1264443781766794 =====================

28. Identifying the trace file name TRACEFILE_IDENTIFIER - Another useful parameter Many a times you need to enable tracing at a session level. And when you do that, you have to make a little bit of effort in digging out your session's trace file. Why? because the trace file is created in the user_dump_dest directory and there would be so many other trace files, and all of them would have similar naming convention "SID_ora_nnnn.trc". However with the help of the parameter "TRACEFILE_IDENTIFIER", you can easily distinguish your trace file. Very useful specifically when you want to use trace analyzer etc. Here is how?1. Set the tracefile identifier as you want, this will be appended to trace file name.alter session set tracefile_identifier=‘filename_sql_trace';2. Enable your session level tracingalter session set sql_trace=true;alter session set events '10046 trace name context forever, level 12' ;3. Execute your queries/statements4. Turn-off tracing alter session set events '10046 trace name context off';alter session set sql_trace=false;5. Check the user_dump_dest folder and you will find a file name "SID_ora_nnnn_filname_sql_trace.trcSee now it's so easy to identify. Having said that you can still find out the trace file name without using TRACEFILE_IDENTIFIER parameter using the following SQL but when Oracle has provided an easier method, why not use it?-- sql to find out trace file name for your session.select c.value || '/' || d.instance_name ||'_ora_' || a.spid || '.trc' tracefrom v$process a, v$session b, v$parameter c, v$instance dwhere a.addr = b.paddrand b.audsid = userenv('sessionid')and c.name = 'user_dump_dest'/

29. What is TKPROF utility TKPROF is a trace kernel profiler – useful for formatting trace files for tuning purposes Flexible options for performance report statistics Cleans up the raw SQL trace files into human readable tuning reports. Oracle Metalink Note: 41634.1 TKPROF Simplistic overview

30. How to use TKPROF TKPROF takes the trace file and output report file plus performance specific parameters Many parameter options for using tkprof Syntax for tkprof: tkprof trace_file.trc output_file.rpt [waits={yes|no}] [sort={yes|no}] [print=option] [aggregate={yes|no}] [insert={filename}] [sys={yes|no}] [table={schema.table}] [explain={schema/password}] [record=filename] [width=n]

31. Getting help with TKPROF TKPROF Help : [oracle@raclinux1 ~]$ tkprof Usage: tkprof tracefile outputfile [explain= ] [table= ] [print= ] [insert= ] [sys= ] [sort= ] table=schema.tablename Use 'schema.tablename' with 'explain=' option. explain=user/password Connect to ORACLE and issue EXPLAIN PLAN. print=integer List only the first 'integer' SQL statements. aggregate=yes|no insert=filename List SQL statements and data inside INSERT statements. sys=no TKPROF does not list SQL statements run as user SYS. record=filename Record non-recursive statements found in the trace file. waits=yes|no Record summary for any wait events found in the trace file. sort=option Set of zero or more of the following sort options: prscnt number of times parse was called prscpu cpu time parsing prsela elapsed time parsing prsdsk number of disk reads during parse prsqry number of buffers for consistent read during parse prscu number of buffers for current read during parse prsmis number of misses in library cache during parse execnt number of execute was called execpu cpu time spent executing exeela elapsed time executing exedsk number of disk reads during execute exeqry number of buffers for consistent read during execute execu number of buffers for current read during execute exerow number of rows processed during execute exemis number of library cache misses during execute fchcnt number of times fetch was called fchcpu cpu time spent fetching fchela elapsed time fetching fchdsk number of disk reads during fetch fchqry number of buffers for consistent read during fetch fchcu number of buffers for current read during fetch fchrow number of rows fetched userid userid of user that parsed the cursor [oracle@raclinux1 ~]$

32. Formatting SQL trace file with TKPROF Example [oracle@raclinux1 trace]$ tkprof RACDB1_ora_9163_products_sqltune.trc tune_sql.rpt waits=yes aggregate=yes sys=yes explain=sys/oracle

33. Output report for SQL Trace with TKPROF ******************************************************************************** SQL ID: gkc6r0xdddpsk Plan Hash: 1954719464 select prod_name, prod_list_price from sh.products where prod_list_price > 7.99 and prod_list_price < 69.99 call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 3 0.00 0.02 0 0 0 0 Execute 4 0.00 0.00 0 0 0 0 Fetch 15 0.00 0.00 0 24 0 156 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 22 0.00 0.03 0 24 0 156 Misses in library cache during parse: 0 Optimizer mode: ALL_ROWS Parsing user id: 88 ********************************************************************************

34. DBMS_SQLTUNE package What is the DBMS_SQLTUNE package? Automates SQL tuning by using profiles to tune SQL. DBMS_SQLTUNE package is used as part of the SQL Tune Advisors (STA) with Oracle 11g

35. Using DBMS_SQLTUNE for Oracle 11g R2 Example SQL> DECLARE sql_task_name VARCHAR2(30); tune_sqltext CLOB; BEGIN tune_sqltext :='select * '||'FROM sh.sales '||' where prod_id=20 and '||' cust_id=200 '; sql_task_name := DBMS_SQLTUNE.CREATE_TUNING_TASK( sql_text=>tune_sqltext, User_name=>'SH', Scope=>'COMPREHENSIVE', Time_limit=>60, Task_name=>'TUNE_SH_TASK', Description=>'New task for tuning SH query'); END; / 2 3 4 5 6 7 8 9 10 11 12 13 14 PL/SQL procedure successfully completed.

36. Using DBMS_SQLTUNE for Oracle 11g R2 (Continued) DBMS_SQLTUNE uses the CREATE_TUNING_TASK function as part of SQL Tuning Advisor (STA) with 11g Once we have created the tuning task, we need to execute it with the DBMS_SQLTUNE.EXECUTE_TUNING_TASK: SQL> begin DBMS_SQLTUNE.EXECUTE_TUNING_TASK(task_name=>'TUNE_SH_TASK'); End; / 2 3 4 PL/SQL procedure successfully completed. SQL>

37. Viewing SQL tuning Advisor Report for 11g R2 To view SQL Tuning report query the DBMS_SQLTUNE.REPORT_TUNING_TASK; SQL> select dbms_sqltune.report_tuning_task(‘TUNE_SH_TASK’) drom DUAL; SQL> / DBMS_SQLTUNE.REPORT_TUNING_TASK('TUNE_SH_TASK') -------------------------------------------------------------------------------- GENERAL INFORMATION SECTION ------------------------------------------------------------------------------- Tuning Task Name : TUNE_SH_TASK Tuning Task Owner : SYS Workload Type : Single SQL Statement Execution Count : 2 Current Execution : EXEC_86 Execution Type : TUNE SQL Scope : COMPREHENSIVE Time Limit(seconds): 60 Completion Status : COMPLETED DBMS_SQLTUNE.REPORT_TUNING_TASK('TUNE_SH_TASK') -------------------------------------------------------------------------------- Started at : 01/26/2010 16:34:26 Completed at : 01/26/2010 16:34:27 ------------------------------------------------------------------------------- Schema Name: SH SQL ID : 8z3nbpj9f0nsa SQL Text : select * FROM sh.sales where prod_id=20 and cust_id=200 ------------------------------------------------------------------------------- FINDINGS SECTION (1 finding) ------------------------------------------------------------------------------- DBMS_SQLTUNE.REPORT_TUNING_TASK('TUNE_SH_TASK') -------------------------------------------------------------------------------- 1- Index Finding (see explain plans section below) -------------------------------------------------- The execution plan of this statement can be improved by creating one or more indices. Recommendation (estimated benefit: 93.13%) ------------------------------------------ - Consider running the Access Advisor to improve the physical schema design or creating the recommended index. If you choose to create the recommended index, consider dropping the index "SH"."SALES_CUST_BIX”

38. Introducing Oradebug What is Oradebug? Why should you use it? Performance Tuning Hang Analysis Dump Analysis for Oracle 11g Memory Suspend and Resume Oracle processes

39. Inside Oradebug Oradebug introduced in version 7.X release of Oracle Poorly documented utility Recommend caution when using oradebug Syntax for Oradebug

40. Help option for Oradebug Oradebug has MANY options!!! Use the oradebug help command to list all the main options available: SQL> oradebug help HELP [command] Describe one or all commands SETMYPID Debug current process SETOSPID <ospid> Set OS pid of process to debug SETORAPID <orapid> ['force'] Set Oracle pid of process to debug SETORAPNAME <orapname> Set Oracle process name to debug SHORT_STACK Get abridged OS stack CURRENT_SQL Get current SQL DUMP <dump_name> <lvl> [addr] Invoke named dump DUMPSGA [bytes] Dump fixed SGA DUMPLIST Print a list of available dumps EVENT <text> Set trace event in process SESSION_EVENT <text> Set trace event in session DUMPVAR <p|s|uga> <name> [level] Print/dump a fixed PGA/SGA/UGA variable DUMPTYPE <address> <type> <count> Print/dump an address with type info SETVAR <p|s|uga> <name> <value> Modify a fixed PGA/SGA/UGA variable PEEK <addr> <len> [level] Print/Dump memory POKE <addr> <len> <value> Modify memory WAKEUP <orapid> Wake up Oracle process SUSPEND Suspend execution RESUME Resume execution FLUSH Flush pending writes to trace file CLOSE_TRACE Close trace file TRACEFILE_NAME Get name of trace file LKDEBUG Invoke global enqueue service debugger NSDBX Invoke CGS name-service debugger -G <Inst-List | def | all> Parallel oradebug command prefix -R <Inst-List | def | all> Parallel oradebug prefix (return output SETINST <instance# .. | all> Set instance list in double quotes SGATOFILE <SGA dump dir> Dump SGA to file; dirname in double quotes DMPCOWSGA <SGA dump dir> Dump & map SGA as COW; dirname in double quotes MAPCOWSGA <SGA dump dir> Map SGA as COW; dirname in double quotes HANGANALYZE [level] [syslevel] Analyze system hang FFBEGIN Flash Freeze the Instance FFDEREGISTER FF deregister instance from cluster FFTERMINST Call exit and terminate instance FFRESUMEINST Resume the flash frozen instance FFSTATUS Flash freeze status of instance SKDSTTPCS <ifname> <ofname> Helps translate PCs to names WATCH <address> <len> <self|exist|all|target> Watch a region of memory DELETE <local|global|target> watchpoint <id> Delete a watchpoint SHOW <local|global|target> watchpoints Show watchpoints DIRECT_ACCESS <set/enable/disable command | select query> Fixed table access CORE Dump core without crashing process IPC Dump ipc information UNLIMIT Unlimit the size of the trace file PROCSTAT Dump process statistics CALL [-t count] <func> [arg1]...[argn] Invoke function with arguments SQL>

41. How to use oradebug – Tracing a session First we need to obtain SPID from v$SESSION and V$PROCESS SQL> select username, sid, serial#, paddr from v$session where username='SH'; USERNAME SID SERIAL# PADDR ------------------------------ ---------- ---------- -------- SH 153 3370 994694F4 SQL> select addr, pid , spid from v$process where addr='994694F4'; ADDR PID SPID -------- ---------- ------------------------ 994694F4 35 20933 SQL>

42. Tracing a session with Oradebug Then we use the SPID in oradebug to trace session: SQL> oradebug setospid 20933 Oracle pid: 35, Unix process pid: 20933, image: oracle@raclinux1.gj.com (TNS V1-V3) SQL> We then set the unlimit parameter for the trace file with Oradebug: SQL> oradebug unlimit Statement processed. SQL> Now we are ready to start the trace for our session with Oradebug SQL> oradebug event 10046 trace name context forever, level 4 Statement processed. SQL>

43. Tracing SQL for session with oradebug Oradebug uses Events for tracing options Some common trace events: 10053 – Cost Based Optimizer (CBO) trace 10046 – SQL Trace To start tracing with Oradebug SQL> oradebug event 10046 trace name context forever, level 4 Statement processed. SQL> To stop and turn off tracing for Oradebug: SQL> oradebug event 10046 trace name context off Statement processed.

44. Locate the session trace file for Oradebug Trace file output based on user_dump_dest parameter Naming convention for the trace file sid_ora_pid_traceid.trc Easy way find trace file is to set trace_identifier parameter: alter session set tracefile_identifier=‘mytrace’; Useful query to find trace file Select u_dump.value||’/’||db_name.value||’_ora_’||v$PROCESS.spid||nvl2(v$process.traceid,’_’||v$process.traceid, Null )||’.trc’ “Trace File” from v$parameter u_dump cross join v$parameter db_name cross join v$process join v$session on v$process.addr = ‘usr_dump_dest’ and db_name.name = ‘db_name’ and v$session.audsid=sys_context(‘userenv’,’sessionid’);

45. Viewing the oradebug trace file for user session Locate the trace file for oradebug Format it with tkprof cp RACDB_1_ora_20933_test_sh_query.trc trc.trc [oracle@raclinux1 trace]$ tkprof trc.trc rpt.rpt aggregate=yes waits=yes sys=no explain=sh/sh TKPROF: Release 11.2.0.1.0 - Development on Tue Jan 26 18:29:25 2010 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. [oracle@raclinux1 trace]$ Open the trace file with text editor $vi rpt.rpt SQL ID: 3j4064f628jmk Plan Hash: 1954719464 select prod_name, prod_list_price from sh.products where prod_list_price > 7.99 and prod_list_price < 69.99 call count cpu elapsed disk query current rows ------- ------ -------- ---------- ---------- ---------- ---------- ---------- Parse 6 0.00 0.00 0 0 0 0 Execute 6 0.00 0.00 0 0 0 0 Fetch 30 0.00 0.00 0 48 0 312 ------- ------ -------- ---------- ---------- ---------- ---------- ---------- total 42 0.00 0.00 0 48 0 312 Misses in library cache during parse: 1 Optimizer mode: ALL_ROWS Parsing user id: 88 (SH) Rows Row Source Operation ------- --------------------------------------------------- 52 TABLE ACCESS FULL PRODUCTS (cr=8 pr=0 pw=0 time=0 us cost=3 size=1820 c ard=52) Rows Execution Plan ------- --------------------------------------------------- 0 SELECT STATEMENT MODE: ALL_ROWS 52 TABLE ACCESS MODE: ANALYZED (FULL) OF 'PRODUCTS' (TABLE)

46. Using oradebug for Hang Analysis Oradebug is excellent tool for solving database hang issue. Use the HANGANALYZE option with oradebug Syntax: oradebug hanganalyze <level> For Oracle RAC environment the syntax is: oradebug –g all hanganalyze <level> Level indicates amount of details to collect for a hang. Refer to Metalink Note: 175006.1 and Note:215858.1

47. Oracle Database Hang analysis with Oradebug The debug levels defined for HANGANALYZE with oradebug are as follows Note 215858.1: Level 10: Dump all process ( IGN state) Level5: Level 4 + Dump all processes involved with chain (NLEAF) Level 4: Level 3 + Dump leaf nodes (blocker) in wait chains (LEAF, LEAF_NW, ING_DMP state) Level 3: Level 2 + Dump only processes considered in hang condition (IN_HANG state) Levels 1-2: Generate ONLY HANGANALYZE output with no process dump

48. Hang Analysis Example for Oracle 11g R2 Set environment for hang analysis: SQL> oradebug setmypid Statement process. SQL> oradebug unlimit Statement processed. Perform Hang Analysis: SQL> oradebug -g all hanganalyze 10 Hang Analysis Dumps in /u01/app/oracle/diag/rdbms/racdb/RACDB_1/trace/RACDB_1_diag_6112.trc SQL>exit

49. Review Hang Analysis Oracle 11g R2 Review output hang analysis from trace file: Trace file /u01/app/oracle/diag/rdbms/racdb/RACDB_1/trace/RACDB_1_diag_6112.trc Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production With the Partitioning, Real Application Clusters, Oracle Label Security, OLAP, Data Mining, Oracle Database Vault and Real Application Testing options ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1 System name: Linux Node name: raclinux1.gj.com Release: 2.6.18-164.el5PAE Version: #1 SMP Thu Sep 3 02:28:20 EDT 2009 Machine: i686 Instance name: RACDB_1 Redo thread mounted by this instance: 0 <none> Oracle process number: 5 Unix process pid: 6112, image: oracle@raclinux1.gj.com (DIAG) *** 2010-01-26 13:02:55.529 *** SESSION ID:(127.1) 2010-01-26 13:02:55.529 *** CLIENT ID:() 2010-01-26 13:02:55.529 *** SERVICE NAME:() 2010-01-26 13:02:55.529 *** MODULE NAME:() 2010-01-26 13:02:55.529 *** ACTION NAME:() 2010-01-26 13:02:55.529 Node id: 0 List of nodes: 0, *** 2010-01-26 13:02:55.565 Reconfiguration starts [incarn=0] *** 2010-01-26 13:02:55.566 I'm the master node *** 2010-01-26 13:02:55.566 Reconfiguration completes [incarn=1] Group reconfiguration cleanup

50. Dump Analysis for Oracle 11g R2 Memory and events Use oradebug to dump contents of UGA and PGA memory Useful for debugging memory leak issues Dump Analysis for Oracle 11g Memory Describe events To start diagnosing a problem with the process size, such as a suspected leak, a heapdump of the offending process is required (Also helps to check with pmap –x <pid> for memory leaks): $sqlplus SQL> connect '/ as sysdba' SQL> oradebug setospid <pid> SQL> oradebug unlimit SQL> oradebug dump heapdump 5 --this dumps PGA and UGA heaps The above information should be provided to Oracle Support Services. Oracle 8i / 9i / 10g SQL> oradebug dump events <level> Level 1 - Dump session group's event settings Level 2 - Dump process group's event settings Level 4 - Dump system group's event settings (Ie the instance wide events) Oracle 10g / 11g and above SQL> oradebug eventdump <level> SESSION - Dump session group's event settings PROCESS - Dump process group's event settings SYSTEM - Dump system group's event settings (Ie the instance wide events) SQL> oradebug setmypid Statement processed. SQL> oradebug eventdump session Statement processed. SQL> oradebug eventdump system Statement processed. SQL> alter system set events '942 trace name errorstack level 3'; System altered. SQL> oradebug eventdump system 942 trace name errorstack level 3 SQL> SQL> oradebug eventdump session 942 trace name errorstack level 3

51. Dump the SGA with Oradebug for Oracle 11g R2 Use the oradebug dumpsga command: SQL> oradebug setmypid Statement processed. SQL> oradebug unlimit Statement processed. SQL> oradebug dumpsga Statement processed. SQL> oradebug tracefile_name /u01/app/oracle/diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_25351.trc SQL> exit

52. Viewing SGA memory dump for Oracle 11g R2 Contents of SGA dump in oradebug trace file Trace file /u01/app/oracle/diag/rdbms/racdb/RACDB_1/trace/RACDB_1_ora_25351.trc Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production With the Partitioning, Real Application Clusters, Automatic Storage Management, Oracle Label Security, OLAP, Data Mining, Oracle Database Vault and Real Application Testing options ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1 System name: Linux Node name: raclinux1.gj.com Release: 2.6.18-164.el5PAE *** SESSION ID:(154.6347) 2010-01-26 19:14:21.438 *** CLIENT ID:() 2010-01-26 19:14:21.438 *** SERVICE NAME:(SYS$USERS) 2010-01-26 19:14:21.438 *** MODULE NAME:(sqlplus@raclinux1.gj.com (TNS V1-V3)) 2010-01-26 19:14:21.438 *** ACTION NAME:() 2010-01-26 19:14:21.438 Processing Oradebug command 'setmypid' *** 2010-01-26 19:14:21.438 Oradebug command 'setmypid' console output: <none> *** 2010-01-26 19:14:30.092 Processing Oradebug command 'unlimit' *** 2010-01-26 19:14:30.092 Oradebug command 'unlimit' console output: <none> *** 2010-01-26 19:15:11.671 ----- Fixed Areas Dump (level=2) ----- ----- Dump of the Fixed SGA ----- ksmsgft ksmsgf_ [20000000, 20001000) = 00000000 00000000 00000000 00000000 ... Dump of memory from 0x20000010 to 0x20001000 20000010 00000000 00000000 00000000 00000000 [................] Repeat 254 times kywmr * kywmrsga_ [20001000, 20001004) = 990CD020

53. Suspend and resume runaway Oracle processe Oradebug useful for stopping and restarting Oracle sessions Be careful when using oradebug in production environments! Syntax for suspending process with oradebug oradebug suspend To resume a process that has been suspended oradebug resume Useful when need to delay batch process for more critical job.

54. Find the session to suspend Find the session information to suspend: SQL> select s.username, s.pid, p.spid From V$session s, v$process p Where s.paddr=p.addr And S.username <> ‘SYS’; USERNAME SID SPID ---------------------- -------- ---------- SH 140 18335

55. Suspend the session with oradebug Oradebug suspend SQL> oradebug setospid 18335 SQL> oradebug suspend Statement processed.

56. Resume process for Oracle 11g R2 with Oradebug Oradebug resume SQL> oradebug resume Statement processed.

57. Using Oradebug for Oracle RAC events oradebug is used to trace issues related to IPC in interconnects and RAC Syntax for IPC memory dump with Oracle RAC: SQL> oradebug ipc

58. Oracle RAC tracing with oradebug for IPC Example SQL> oradebug setmypid Statement processed. SQL>oradebug unlimit Statement processed. SQL>oradebugipc Information written to trace file. SQL> oradebug trace_file /u01/app/oracle/admin/RACDB/udump/racdb1_ora_6391.trc SQL>exit

59. Oracle11g R2 RAC trace with oradebug Review trace file for interconnect trace with Oradebug for RAC: SSKGXPT 0xcc75e9c flags SSKGXPT_READPENDING info for network 0 socket no 7 IP 10.10.10.11 UDP 2247 sflagsSSKGXPT_UP info for network 1 socket no 0 IP 0.0.0.0 UDP 0 sflagsSSKGXPT_DOWN active 0 actcnt1 context timestamp 0 no ports sconnoacconoerttstateseq# sent asyncsyncrtransacks

60. Hidden Oracle 11g R2 parameter Oracle 11g has many new additional hidden parameters! There are more than 2050 hidden parametrs for Oracle 11g R2! SQL> select count(*) from x$ksppi where substr(ksppinm,1,1)='_'; COUNT(*) ---------- 2057 SQL> Query to find out these hidden Oracle 11g R2 parameters: Select ksppinm from x$ksppi where substr(ksppinm,1,1)=‘_’;

61. Some new hidden parameters for Oracle 11g R2 Some hidden parameters for 11g new features Hang manager hidden parameters for Oracle 11g R2: _hm_analysis_ouput_disk _hm_analysis_oradebug_node_dump_level _hm_analysis_oradebug_sys_dump_level _hm_verification_interval _hm_log_incidents Be careful using these First change in test and sandbox Best to work with Oracle support since those are undocumented parameters!

62. Oracle 11g V$ and X$ tables for troubleshooting and tuning 11g R2 Each release of Oracle adds many new V$ and X$ performance view tables. More than 945 different X$ tables in Oracle 11g R2. There are over 525 different v$ performance views in Oracle 11g R2 . Query to find the V$ performance views for Oracle 11g R2: Select name, type from v$fixed_table where name like ‘V$%’; Query to find the list of X$ tables for Oracle 11g R2: Select name, type from v$fixed_tablewhere name like ‘X$%’;

63. Useful v$ and X$ table for Oracle 11g R2 Some interesting new V$ Dynamic Performance Views V$MEMORY_DYNAMIC_COMPONENTS - displays information about the dynamic SGA components. This view summarizes information based on all completed SGA resize operations since instance startup. All sizes are expressed in bytes. V$MEMORY_TARGET_ADVICE - information about how the MEMORY_TARGET parameter should be sized based on current sizing and satisfaction metrics. V$ALERT_TYPES – information for server alert types V$IR_FAILURE – provides information on the failure. Note that records in this view can have parent records within this view. (DRA) V$IR_REPAIR – show the results of repair failure commands. (DRA) V$IR_MANUAL_CHECKLIST – detailed information messages related to the failure. Messages provide information on how to manually correct the problem. (DRA) V$IR_FAILURE_SET – list of advice records associated with the failure. (DRA) V$HM_CHECK - information about all the checks registered with Health Monitor. Each check is uniquely identified by a name or an ID. V$HM_CHECK_PARAM - information about the input parameters of all Health Monitor checks. V$HM_RUN - information about all Health Monitor checks and their status. All checks ran. V$HM_FINDING - information about all the findings of various Health Monitor runs. V$HM_RECOMMENDATION - information about any recommendation from Health check runs. V$HM_INFO - Information about runs, finding and recommendations. Perform a describe on each of these V$ dynamic performance views for detail information.

64. Some useful X$ tables for Oracle 11g R2 These are some useful new X$ tables for Oracle 11g R2: X$DIAG_INFO X$IR_WORKING_FAILURE_SET – incident reporting X$IR_WORKING_REPAIR_SET – incident reporting X$IR_REPAIR_OPTION – incident reporting X$IR_MANUAL_OPTION – incident reporting X$IR_REPAIR_STEP – incident reporting

65. Power tools for UNIX and Windows When Oracle tools are not enough, these OS tools are essential: dbx (Solaris and Linux) gdb (Linux) similar to dbx truss (Solaris) strace (Linux) orakill (Windows) STACKX (UNIX) Metalink Note: 362791.1 Useful References on Metalink for core Debug Analysis: Oracle Metalink Note:16534.1 “debug Core Dumps / core files on Unix” Oracle Metalink Note: 1007808.6 “How to handle core dumps on UNIX”

66. DBX – Debugging Solaris and Linux DBX is trace debugger for Solaris and also available for free download from Sun.com as part of Solaris Developer Studio. Provides comprehensive debugger level commands for Solaris and Linux OS systems calls and Oracle 11g instance and network level internal processes. DBX is an advanced complex tool with complex syntax. Complete reference guide from SUN: Sun Studio 12: Debugging a Program with dbx: http:/docs.sun.com/app/docs

67. Using dbx for Oracle 11g R2 Troubleshooting DBX can be used to trace Oracle 11g processes Getting help with dbx [root@raclinux1 bin]# ./dbx For information about new features see `help changes' To remove this message, put `dbxenv suppress_startup_message 7.7' in your .dbxrc (dbx) help dbx: warning: Help file from 'C' locale is used. No help file in the locale 'en_US.UTF-8' dbx: warning: Help file from 'C' locale is used. No help file in the locale 'en_US.UTF-8' command summary Use `commands' to see a command summary consisting of one-line descriptions of each dbx command. Execution and Tracing cancel catch clear cont delete fix fixed handler ignore intercept next pop replay rerun restore run runargs save status step stop trace unintercept when whocatches Displaying and Naming Data assign call dis display down dump examine exists fortran_modules frame hide print undisplay unhide up watch whatis where whereami whereis which Accessing Source Files bsearch cd edit file files func funcs line list loadobject loadobjects module modules pathmap pwd search Debugging Multiple Threads lwp lwps sync syncs thread threads

68. Using dbx for Oracle 11g R2 Find the PID for Oracle process [root@raclinux1 bin]# ps -ef | grep oracleRACDB oracle 9546 9545 0 18:26 ? 00:00:00 oracleRACDB_1 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq))) root 15103 12453 0 20:02 pts/3 00:00:00 grep oracleRACDB [root@raclinux1 bin]# Now we need to get a trace for Oracle [root@raclinux1 bin]# ./dbx /u01/app/oracle/product/11.2.0/db_1/bin/oracle 9546 For information about new features see `help changes' To remove this message, put `dbxenv suppress_startup_message 7.7' in your .dbxrc Reading oracle Reading ld-linux.so.2 …………………………… Reading libnque11.so Attached to process 9546 t@3671744 (l@9546) stopped in (unknown) at 0xdcd410 0x00dcd410: popl %ebp (dbx) Enable Collection Process and continue trace - The collector command collects performance data for analysis by the Performance Analyzer. It is valid only in native mode. (dbx) collector enable Back to our DBX trace session: (dbx) cont

69. Continued – Tracing Oracle 11g R2 with dbx Use the dbx thread command to look at Oracle 11g R2 memory process stack. Use the dbx thread command. Displays the current thread. (dbx) thread t@3671744 t@3671744 (l@9546) stopped in (unknown) at 0xdcd410 0x00dcd410: popl %ebp (dbx) where current thread: t@3671744 [1] 0xdcd410(0x2010, 0x108ba596, 0x7cb763, 0x105d4874, 0xf4351a0, 0x9), at 0xdcd410 [2] __read_nocancel(0x108b1108, 0x108b1770, 0x108ae8f8, 0x108ba596, 0xbfaa3d10, 0x0), at 0x7cb763 [3] ntpfprd(0x108ae790, 0x108ba596, 0x2010, 0xbfaa3e24, 0x8), at 0xf4350d9 [4] nsbasic_brc(0x108a5c48, 0x108a5d64, 0xbfaa3fa4, 0x0), at 0xf41b0b7 [5] nsbrecv(0x108a5c48, 0x108a5d64, 0xbfaa3fa4, 0x0), at 0xf41dba6 [6] nioqrc(0x106dad04, 0x0, 0xbfaa5f0c, 0x1, 0x0, 0x1), at 0xf422429 [7] __PGOSF25_opikndf2(0x106dad04, 0x2, 0xbfaa5f0c, 0x1, 0x0), at 0xf1c88e6 [8] opitsk(0x0, 0x0), at 0x8f3b9fb [9] opiino(0x3c, 0x4, 0xbfaa6e14), at 0x8f3f89b [10] opiodr(0x3c, 0x4, 0xbfaa6e14, 0x0), at 0xf1cb036 [11] opidrv(0x3c, 0x4, 0xbfaa6e14), at 0x8f382f3 [12] sou2o(0xbfaa6df8, 0x3c, 0x4, 0xbfaa6e14), at 0x92ed66c [13] opimai_real(0x2, 0xbfaa6f48, 0x108721a8, 0x0, 0x0, 0x651f14), at 0x8542cdb [14] ssthrdmain(0x2, 0xbfaa6f48), at 0x92f1f2d [15] main(0x2, 0xbfaa7044, 0xbfaa7050, 0x640810, 0x0, 0x1), at 0x8542c57 (dbx) (dbx) thread -info Thread t@3671744 (0x0) at priority 0 state: active on l@9546 base function: 0x0: 0x00000000() stack: 0x0[0] flags: (none) masked signals: (none) Currently active at 0x00dcd410 (dbx)

70. Debug Oracle 11g R2 with dbx The debug function with dbx is useful for Oracle 11g R2(debug lists the program being debugged) : (dbx) debug –f Debugging: oracle ( process id 24976)

71. Truss: Solaris powerful tool for tracing Oracle 11g R2 Truss provides ability to walk through trace at step by step level for Oracle 11g internal process Syntax: truss [-fcaeil ] [-[tvx] [!]syscall..] [-s [!] signal..] [-m [!]fault..] [-[rw] [!]fd..] [-o outfile] command | -p pid Typical options for truss: -o output to file -f trace child processes -c count system calls -p trace calls based on Unix pid

72. How to use truss with Solaris for Oracle 11g R2 Example solaris02$truss –cp 3316 ^C Syscall seconds calls errors Read .000 2 Write .000 2 Times .000 33 Yield .000 226

73. Strace tool for Linux and Oracle 11g R2 tracing Strace is similar to truss for Linux O/S tracing with Oracle 11g Powerful tool with command line syntax: usage: strace [-dffhiqrtttTvVxx] [-a column] [-e expr] ... [-o file] [-p pid] ... [-s strsize] [-u username] [-E var=val] ... [command [arg ...]] or: strace -c [-e expr] ... [-O overhead] [-S sortby] [-E var=val] ... [command [arg ...]]

74. How to use strace with Linux and Oracle 11g R2 Options available for strace [root@raclinux1 ~]# strace usage: strace [-dffhiqrtttTvVxx] [-a column] [-e expr] ... [-o file] [-p pid] ... [-s strsize] [-u username] [-E var=val] ... [command [arg ...]] or: strace -c [-e expr] ... [-O overhead] [-S sortby] [-E var=val] ... [command [arg ...]] -c -- count time, calls, and errors for each syscall and report summary -f -- follow forks, -ff -- with output into separate files -F -- attempt to follow vforks, -h -- print help message -i -- print instruction pointer at time of syscall -q -- suppress messages about attaching, detaching, etc. -r -- print relative timestamp, -t -- absolute timestamp, -tt -- with usecs -T -- print time spent in each syscall, -V -- print version -v -- verbose mode: print unabbreviated argv, stat, termio[s], etc. args -x -- print non-ascii strings in hex, -xx -- print all strings in hex -a column -- alignment COLUMN for printing syscall results (default 40) -e expr -- a qualifying expression: option=[!]all or option=[!]val1[,val2]... options: trace, abbrev, verbose, raw, signal, read, or write -o file -- send trace output to FILE instead of stderr -O overhead -- set overhead for tracing syscalls to OVERHEAD usecs -p pid -- trace process with process id PID, may be repeated -s strsize -- limit length of print strings to STRSIZE chars (default 32) -S sortby -- sort syscall counts by: time, calls, name, nothing (default time) -u username -- run command as username handling setuid and/or setgid -E var=val -- put var=val in the environment for command -E var -- remove var from the environment for command [root@raclinux1 ~]#

75. Example for strace for Oracle 11g R2 Perform system call trace for Oracle 11g: [oracle@raclinux1 ~]$ strace oracle execve("/u01/app/oracle/product/11.2.0/db_1/bin/oracle", ["oracle"], [/* 36 vars */]) = 0 brk(0) = 0x10792000 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x8b9000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/u01/app/oracle/product/11.2.0/db_1/lib/tls/i686/sse2/libodm11.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/u01/app/oracle/product/11.2.0/db_1/lib/tls/i686/sse2", 0xbfa7b348) = -1 ENOENT (No such file or directory) open("/u01/app/oracle/product/11.2.0/db_1/lib/tls/i686/libodm11.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/u01/app/oracle/product/11.2.0/db_1/lib/tls/i686", 0xbfa7b348) = -1 ENOENT (No such file or directory) open("/u01/app/oracle/product/11.2.0/db_1/lib/tls/sse2/libodm11.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/u01/app/oracle/product/11.2.0/db_1/lib/tls/sse2", 0xbfa7b348) = -1 ENOENT (No such file or directory) open("/u01/app/oracle/product/11.2.0/db_1/lib/tls/libodm11.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/u01/app/oracle/product/11.2.0/db_1/lib/tls", 0xbfa7b348) = -1 ENOENT (No such file or directory) open("/u01/app/oracle/product/11.2.0/db_1/lib/i686/sse2/libodm11.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/u01/app/oracle/product/11.2.0/db_1/lib/i686/sse2", 0xbfa7b348) = -1 ENOENT (No such file or directory) open("/u01/app/oracle/product/11.2.0/db_1/lib/i686/libodm11.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/u01/app/oracle/product/11.2.0/db_1/lib/i686", 0xbfa7b348) = -1 ENOENT (No such file or directory) open("/u01/app/oracle/product/11.2.0/db_1/lib/sse2/libodm11.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/u01/app/oracle/product/11.2.0/db_1/lib/sse2", 0xbfa7b348) = -1 ENOENT (No such file or directory) open("/u01/app/oracle/product/11.2.0/db_1/lib/libodm11.so", O_RDONLY) = 3 read(3, "77ELF54004"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0644, st_size=8857, ...}) = 0 mmap2(NULL, 6768, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40d000 mmap2(0x40e000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x40e000 close(3) = 0 open("/u01/app/oracle/product/11.2.0/db_1/lib/libcell11.so", O_RDONLY) = 3 read(3, "77ELF40|004"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0644, st_size=383547, ...}) = 0

76. Tracing System calls with strace with Oracle 11g R2 Similar to truss: Check for system call errors for Oracle 11g with strace [oracle@raclinux1 ~]$ strace -c oracle % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 40.53 0.000308 308 1 execve 15.79 0.000120 0 338 49 open 15.13 0.000115 7 16 16 mkdir 12.37 0.000094 1 75 read 8.29 0.000063 0 132 munmap 7.89 0.000060 0 293 close 0.00 0.000000 0 20 write 0.00 0.000000 0 1 chmod 0.00 0.000000 0 3 lseek 0.00 0.000000 0 3 3 access 0.00 0.000000 0 1 dup 0.00 0.000000 0 4 times 0.00 0.000000 0 4 brk 0.00 0.000000 0 2 setrlimit 0.00 0.000000 0 5 gettimeofday 0.00 0.000000 0 5 uname 0.00 0.000000 0 9 mprotect 0.00 0.000000 0 3 _llseek 0.00 0.000000 0 6 rt_sigaction 0.00 0.000000 0 5 rt_sigprocmask 0.00 0.000000 0 5 getrlimit 0.00 0.000000 0 307 mmap2 0.00 0.000000 0 48 19 stat64 0.00 0.000000 0 7 2 lstat64 0.00 0.000000 0 26 fstat64 0.00 0.000000 0 2 getuid32 0.00 0.000000 0 1 geteuid32 0.00 0.000000 0 6 fcntl64 0.00 0.000000 0 1 gettid 0.00 0.000000 0 2 futex 0.00 0.000000 0 1 set_thread_area 0.00 0.000000 0 1 set_tid_address 0.00 0.000000 0 1 set_robust_list 0.00 0.000000 0 3 socket 0.00 0.000000 0 1 bind 0.00 0.000000 0 2 2 connect 0.00 0.000000 0 5 shmat 0.00 0.000000 0 5 shmdt 0.00 0.000000 0 12 9 shmget ------ ----------- ----------- --------- --------- ---------------- 100.00 0.000760 1362 100 total [oracle@raclinux1 ~]$

77. Tracing system calls for Oracle 11g R2 DBWR Example : using strace to find system calls for Oracle DBWR process oracle 6123 1 0 12:16 ? 00:00:01 ora_dbw0_RACDB_1 oracle 31050 8191 0 20:24 pts/2 00:00:00 grep dbw [oracle@raclinux1 admin]$ strace -c -p 6123 Process 6123 attached - interrupt to quit Process 6123 detached % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 57.30 0.049795 732 68 io_submit 35.52 0.030866 123 250 pread64 3.14 0.002729 0 20064 getrusage 1.55 0.001350 11 123 io_getevents 1.15 0.000999 4 235 125 semtimedop 0.34 0.000297 2 151 semctl 0.32 0.000279 0 17913 gettimeofday 0.25 0.000219 4 58 mmap2 0.25 0.000214 4 50 sendmsg 0.10 0.000087 2 57 open 0.08 0.000068 1 57 read 0.00 0.000000 0 57 close 0.00 0.000000 0 766 times ------ ----------- ----------- --------- --------- ---------------- 100.00 0.086903 39849 125 total [oracle@raclinux1 admin]$

78. Using orakill in Windows and Oracle 11g R2 Useful tool available for Oracle 11g on Windows called orakill Oracle Metalink Note: 69882.1 Useful for killing runaway processes on Oracle 11g on Windows as well as for zombie and hung sessions Syntax: orakill <sid> ,<thread> Can use QSLICE.exe to find threads ( available on Windows NT/200X Resource kit from Microsoft)

79. How to use orakill for windows and Oracle 11g R2 Query if not using QSLICE.EXE for details: C:sersser>orakill Usage: orakill sid thread where sid = the Oracle instance to target thread = the thread id of the thread to kill The thread id should be retrieved from the spid column of a query such as: select spid, osuser, s.program from v$process p, v$session s where p.addr=s.paddr; C:sersser>

80. Example: Using orakill for Windows and Oracle 11g R2 Qslise.exe output Use Qslice to find Oracle processes Than use orakill to end process that is hung C:> orakill ora11g 985 Kill of thread id 985 in instance ora11g successfully signaled.

81. ORA-0600: Tips and Tricks ORA-0600: Internal errors Call Oracle support first Leverage Metalink ORA-0600 support matrix Collect dumps with oradebug if possible

82. What is ORA-0600 error ? Internal Oracle errors generate kernel memory dump. Dump core file and trace files. Useful Metalink Notes: 146580.1, 146581.1 Use the Oracle Metalink ORA-0600 lookup tool for addressing these internal errors. Note 153788.1 – Troubleshoot an ORA-0600 Error using the ORA-600 Argument lookup tool

83. Using the ORA-0600 lookup tool From Oracle support see Note 153788.1 Troubleshoot an ORA-600 Error using the ORA-600 argument lookup tool Can also be used to review and analyze ORA-7445 errors

84. Resolving ORA-0600 errors Work closely with Oracle Support Usually caused by bug such as memory leak Test recent changes between development, test, QA and production to ensure continuity of issue. Ideal to have same release of Oracle to be able to reproduce the error for all environments and platforms.

85. ORA-0600 troubleshooting Use ADR to review incident and associated trace file Dump continued from file: /u01/app/oracle/diag/rdbms/ora11g/ORA11G/trace/ORA11G_ora_12562.trc ORA-00600: internal error code, arguments: [kssadpm: null parent], [], [], [], [], [], [], [] Plug in the arguments from trace file to ORA-0600 Metalink troubleshooting lookup tool. The related solution will then be displayed: Known Issues: Bug# 5605370 * See Note 5605370.8* This bug is alerted in Note 454464.1Various dumps / instance crash possible Fixed: 10.2.0.4, 11.1.0.6

86. KFOD Utility in 11g R2 KFOD for ASM monitoring KFOD help=y [oracle@raclinux1 ~]$ kfod help=y _asm_a/llow_only_raw_disks KFOD allow only raw devices [_asm_allow_only_raw_disks=TRUE/(FALSE)] _asm_l/ibraries ASM Libraries[_asm_libraries=lib1,lib2,...] _asms/id ASM Instance[_asmsid=sid] a/sm_diskstring ASM Diskstring [asm_diskstring=discoverystring, discoverystring ...] c/luster KFOD cluster [cluster=TRUE/(FALSE)] db/_unique_name db_unique_name for ASM instance[db_unique_name=dbname] di/sks Disks to discover [disks=raw,asm,all] ds/cvgroup Include group name [dscvgroup=TRUE/(FALSE)] g/roup Disks in diskgroup [group=diskgroup] h/ostlist hostlist[hostlist=host1,host2,...] metadata_a/usize AU Size for Metadata Size Calculation metadata_c/lients Client Count for Metadata Size Calculation metadata_d/isks Disk Count for Metadata Size Calculation metadata_n/odes Node Count for Metadata Size Calculation metadata_r/edundancy Redundancy for Metadata Size Calculation n/ohdr KFOD header suppression [nohdr=TRUE/(FALSE)] o/p KFOD options type [OP=DISKS/CANDIDATES/MISSING/GROUPS/INSTS/VERSION/CLIENTS/RM/RMVERS/DFLTDSTR/GPNPDSTR/METADATA/ALL] p/file ASM parameter file [pfile=parameterfile] s/tatus Include disk header status [status=TRUE/(FALSE)] v/erbose KFOD verbose errors [verbose=TRUE/(FALSE)] KFOD-01000: USAGE: kfod op=<op> asm_diskstring=... | pfile=...

87. KFOD utility 11g R2 [oracle@raclinux1 ~]$ kfod disk=all -------------------------------------------------------------------------------- Disk Size Path User Group ================================================================================ 1: 15358 Mb /dev/sdb1 oracle oinstall 2: 15358 Mb /dev/sdc1 oracle oinstall 3: 15358 Mb /dev/sdd1 oracle oinstall 4: 40954 Mb /dev/sde1 oracle oinstall -------------------------------------------------------------------------------- ORACLE_SID ORACLE_HOME ================================================================================ +ASM1 /u01/app/11.2.0/grid [oracle@raclinux1 ~]$ kfod op=groups -------------------------------------------------------------------------------- Group Size Free Redundancy Name ================================================================================ 1: 40954 Mb 30440 Mb EXTERN FLASH 2: 46074 Mb 42080 Mb NORMAL DATA [oracle@raclinux1 ~]$ [oracle@raclinux1 ~]$ kfod op=insts -------------------------------------------------------------------------------- ORACLE_SID ORACLE_HOME ================================================================================ +ASM1 /u01/app/11.2.0/grid [oracle@raclinux1 ~]$ [oracle@raclinux1 ~]$ kfod op=version -------------------------------------------------------------------------------- ORACLE_SID RAC VERSION ================================================================================ +ASM1 YES 11.2.0.1.0 [oracle@raclinux1 ~]$ kfod op=clients -------------------------------------------------------------------------------- ORACLE_SID VERSION ================================================================================ RACDB_1 11.2.0.1.0 +ASM1 11.2.0.1.0 +ASM1 11.2.0.1.0 RACDB_1 11.2.0.1.0 [oracle@raclinux1 ~]$ [oracle@raclinux1 ~]$ kfod op=rm -------------------------------------------------------------------------------- Rolling Migration State ================================================================================ Inactive [oracle@raclinux1 ~]$ kfod op=rmvers -------------------------------------------------------------------------------- Rolling Migration Compatible Versions ================================================================================ 11.1.0.6.0 11.1.0.7.0 [oracle@raclinux1 ~]$

88. Questions & Answers Thank you for attending Fill out your evaluation Questions & Answers