One of the biggest challenges when maintaining Oracle software, is the number of components to be considered and the corresponding number of trace files. This is especially the case if you are operating Grid Infrastructures because there are additional steps of complexity added. Analysing Real Application Clusters with two or more nodes involves additional analysis due to increasing number of additional trace logs. This session introduces you to Trace File Analyzer Collector ("TFACTL"), a CLI tool, which integrates several analysis tools to monitor various GI/DB components even on multiple nodes.

1. BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENF
HAMBURG KOPENHAGEN LAUSANNE MÜNCHEN STUTTGART WIEN ZÜRICH
TechEvent 09/2016
Analyzing Oracle related issues using TFA Collector
Rainer Klomps

2. Agenda
Analyzing Oracle related issues using TFA Collector2 09.09.2016
1. What is TFA Collector?
2. Installation / Upgrade
3. Interface
4. Settings
5. Security
6. Analysis
7. Data Redaction
8. Some Samples
9. References

3. Analyzing Oracle related issues using TFA Collector3 09.09.2016
What is TFA Collector?

4. What is TFA Collector?
Analyzing Oracle related issues using TFA Collector4 09.09.2016
Oracle provided diagnostic tool
Acronym for Trace File Analyzer Collector
Collects Trace Files of
– CRS
– ASM
– RDBMS
– Exadata storage server
– User specific trace- and logfiles (configurable)
Stores tracefiles in a Berkeley DB
Very flexible concerning issue analysis
Current version 12.1.2.8.0

5. Architecture (per Node)
What is TFA Collector?
Analyzing Oracle related issues using TFA Collector5 09.09.2016
TFA Daemon
„TFAMain“ process
JAVA VM
TFA CLI
„tfactl“
Berkeley DB
Secure Socket
Log-/Trace Files
(DB, ASM, CRS,
ACFS, OS, Exadata
storage server,
user defined files)

6. Supported plattforms (TFA Collector 12.1.2)
– Intel Linux (Enterprise Linux, RedHat, SUSE)
– Linux Itanium
– Oracle Solaris SPARC
– Oracle Solaris x86-64
– AIX
– HPUX Itanium
– HPUX PA-RISC
– zLinux
What is TFA Collector?
Analyzing Oracle related issues using TFA Collector6 09.09.2016

7. Analyzing Oracle related issues using TFA Collector7 09.09.2016
Installation / Upgrade

8. Installation
Analyzing Oracle related issues using TFA Collector8 09.09.2016
Requirements
– bash 3.2 or higher
– JRE 1.5 or later
– In RAC installations: JRE in same directory on each node
– root access
– for automatic clusterwide installation: passwordless ssh connection via root user

9. Installation
Analyzing Oracle related issues using TFA Collector9 09.09.2016
Allready installed on ODA
Already installed with GI
Unpack TFALite_v<version#>.zip, contents are
– self extracting Library ‘installTFALite‘ (1.500 Lines of Code + Library extracted by
AWK)
– User Guide TFACollectorDocV<version#>.pdf
Execute installTFALite either in silent or in interactive mode
In RAC environment node information comes from CRS, provided CRS stack is up

10. Installation
Analyzing Oracle related issues using TFA Collector10 09.09.2016
options offered with installTFALite
– Without any parameter  interview process
– -local
– -deferdiscovery
– -tfabase (installTFALite installs into given <directory> or <directory>/tfa
– -javahome (alternatively set environment variable JAVA_HOME)
– -silent
– -debug

11. Installation
Analyzing Oracle related issues using TFA Collector11 09.09.2016
<Live demonstration>

12. Upgrade
Analyzing Oracle related issues using TFA Collector12 09.09.2016
Upgrade is as easy as installation
– Just execute the installation procedure installTFALite
– TFA recognizes a previous installation and upgrades it automatically

13. Analyzing Oracle related issues using TFA Collector13 09.09.2016
Interface

14. Interface
Analyzing Oracle related issues using TFA Collector14 09.09.2016
tfactl shell / tfactl command
– provides some support tools
– provides lots of options to control sampling and analysis
Help concerning command syntax
– tfactl –h

15. Interface / Support Tools
Analyzing Oracle related issues using TFA Collector15 09.09.2016
tfactl> toolstatus
.---------------------------------.
| External Support Tools |
+------+---------------+----------+
| Host | Tool | Status |
+------+---------------+----------+
| node1 | alertsummary | DEPLOYED |
| node1 | ls | DEPLOYED |
| node1 | pstack | DEPLOYED |
| node1 | orachk | DEPLOYED |
| node1 | sqlt | DEPLOYED |
| node1 | grep | DEPLOYED |
| node1 | summary | DEPLOYED |
| node1 | prw | RUNNING | (=ProcWatcher)
| node1 | vi | DEPLOYED |
| node1 | tail | DEPLOYED |

16. Interface / Support Tools
Analyzing Oracle related issues using TFA Collector16 09.09.2016
tfactl> toolstatus
.---------------------------------.
| External Support Tools |
+------+---------------+----------+
| Host | Tool | Status |
+------+---------------+----------+
| node1| param | DEPLOYED |
| node1| dbglevel | DEPLOYED |
| node1| darda | DEPLOYED | (=Diagnostic Assistant/RDA)
| node1| history | DEPLOYED |
| node1| oratop | DEPLOYED |
| node1| oswbb | DEPLOYED |
| node1| changes | DEPLOYED |
| node1| events | DEPLOYED |
| node1| ps | DEPLOYED |
'------+---------------+----------'

17. Interface / Options
Analyzing Oracle related issues using TFA Collector17 09.09.2016
Help concerning command syntax
– tfactl –h
enable / disable automatic start on failure / reboot
– tfactl enable / disable
start / stop the tfa daemon
– tfactl start / stop
List events summary and search strings in logfiles
– tfactl analyze

18. Interface / Options
Analyzing Oracle related issues using TFA Collector18 09.09.2016
Print status
– tfactl print status
.-----------------------------------------------------------------------------------------------.
| Host | Status of TFA | PID | Port | Version | Build ID | Inventory Status |
+----------+---------------+------+------+------------+----------------------+------------------+
| training | RUNNING | 2451 | 5000 | 12.1.2.8.0 | 12128020160623222219 | COMPLETE |
'----------+---------------+------+------+------------+----------------------+------------------'

19. Interface / Options
Analyzing Oracle related issues using TFA Collector19 09.09.2016
Print directories being observed for log and trace files
– tfactl print directories (two sample directories displayed)
.------------------------------------------------------------------------------------.
| training |
+------------------------------------+-----------------------+------------+----------+
| Trace Directory | Component | Permission | Added By |
+------------------------------------+-----------------------+------------+----------+
| /u00/app/oracle/diag/rdbms/tvd12cd | [RDBMS]|instance=…, | public | root |
| b/TVD12CDB/cdump | RDBMS|database=… | | |
| Collection policy : No Exclusions | | | |
+------------------------------------+-----------------------+------------+----------+
| /u00/app/oraInventory/logs | [INSTALL] | public | root |
| Collection policy : Exclusions | | | |

20. Interface / Options
Analyzing Oracle related issues using TFA Collector20 09.09.2016
Print information about the BDB repository
– tfactl print repository
.---------------------------------------------------------------.
| training |
+----------------------+----------------------------------------+
| Repository Parameter | Value |
+----------------------+----------------------------------------+
| Location | /u00/app/oracle/product/tfa/repository |
| Maximum Size (MB) | 2555 |
| Current Size (MB) | 135 |
| Free Size (MB) | 2420 |
| Status | OPEN |
'----------------------+----------------------------------------'

21. Analyzing Oracle related issues using TFA Collector21 09.09.2016
Settings

22. Settings
Analyzing Oracle related issues using TFA Collector22 09.09.2016
Configuration parameter Value Default
TFA Version 12.1.2.8.0 --
Automatic diagnostic collection ON OFF
Alert Log Scan ON ON
Trimming of files during diagcollection ON ON
Repository current size (MB) in node1 526 --
Repository maximum size (MB) in node1 10240 Min( 10 G, 50% free space)
Inventory Trace Level 1 1
Collection Trace level 1 1
tfactl> tfactl print config
Configuration parameters and sample settings (1)

23. Settings
Analyzing Oracle related issues using TFA Collector23 09.09.2016
Configuration parameter Value Default
Scan Trace level 1 1
Other Trace level 1 1
Max Size of TFA Log (MB) 50 50
Max Number of TFA Logs 10 10
Max Size of Core File (MB) 20 20
Max Collection Size of Core Files (MB) 200 200
Automatic Purging ON ON
Minimum Age of Collections to Purge (Hours) 12 12
Minimum Space Free to enable Alert Log Scan (MB) 500 500
Configuration parameters and sample settings (2)
 Future, not yet implemented

24. Analyzing Oracle related issues using TFA Collector24 09.09.2016
Security

25. Security
Analyzing Oracle related issues using TFA Collector25 09.09.2016
Root for installation, configuration and analysis
Grant access to tfactl via sudo
Some non risk commands are available for public
Oracle home owners and OS DBA groups are initially setup to use tfactl
– tfactl access
• lsusers
• add, remove user/group
• block, unblock

26. Security
Analyzing Oracle related issues using TFA Collector26 09.09.2016
– tfactl access (continued)
• enable, disable
• reset
• removeall

27. Analyzing Oracle related issues using TFA Collector27 09.09.2016
Analysis

28. Analysis
Analyzing Oracle related issues using TFA Collector28 09.09.2016
tfactl analyze
– -search pattern
• -comp {DB; ASM;CRS;ACFS;OS;OSW;OSWSLABINFO}
(OSW=OSWatcher
OSWLABINFO = information about memory management}
• -type {ERROR; WARNING; generic}
• -node {local | <comma separated list of nodes>}
• -since <duration before current time>

29. Analysis
Analyzing Oracle related issues using TFA Collector29 09.09.2016
tfactl analyze
• -for <specified time>
• -from <specified time> -to <specified time>
• -verbose
• -o <output file>

30. Analyzing Oracle related issues using TFA Collector30 09.09.2016
Data Redaction

31. Data Redaction
Analyzing Oracle related issues using TFA Collector31 09.09.2016
File <TFA_HOME>/resources/mask_strings.xml
<mask_strings>
<mask_string>
<original>top_secret_pattern_1</original>
<replacement>replaced_1</replacement>
</mask_string>
<mask_string>
<original>top_secret_pattern_2</original>
<replacement>replaced_2</replacement>
</mask_string>
…
…
…
</mask_strings>

32. Analyzing Oracle related issues using TFA Collector32 09.09.2016
Some Samples

33. Some Samples
Analyzing Oracle related issues using TFA Collector33 09.09.2016
tfactl> alertsummary
…
Aug 22 16:06:46 Database started
Summary: Ora-600=0, Ora-7445=3, Ora-700=0
~~~~~~~
Warning: Only FATAL errors reported
Warning: These errors were seen and NOT reported
Ora-00313 Ora-00312 Ora-27037 Ora-01157 Ora-01110 Ora-01186 Ora-00060
Ora-01653 Ora-27072 Ora-00227 Ora-00202 Ora-00210 Ora-27048 Ora-01565
Ora-01652 Ora-19815 Ora-19809 Ora-19804 Ora-12012 Ora-01089 Ora19502
Reading /u00/app/oracle/diag/rdbms/tvd12cdb/TVD12CDB/trace/alert_TVD12CDB.log
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
-------------------------------------------------------------------------
Aug 01 20:32:27 Database started

34. Some Samples
Analyzing Oracle related issues using TFA Collector34 09.09.2016
tfactl> changes
Aug/22/2016 16:11:31 : Parameter 'fs.aio-nr' value changed : 18688 => 20608
Aug/22/2016 16:11:31 : Parameter 'fs.dentry-state' value changed: …
Aug/22/2016 16:11:31 : Parameter 'fs.file-nr' value changed : …
Aug/22/2016 16:11:31 : Parameter 'fs.inode-nr' value changed : …
Aug/22/2016 16:11:31 : Parameter 'fs.inode-state' value changed : 36349 10400 0 0 0
Aug/22/2016 16:11:31 : Parameter
'TVD12.training.trivadistraining.com.TVD12.TVD12.shared_pool_reserved_size' value
changed : 15938355 => 17196646

35. Some Samples
Analyzing Oracle related issues using TFA Collector35 09.09.2016
tfactl> ls alert
Output from host : training
------------------------------
/u00/app/oracle/diag/rdbms/tvd12_site1/TVD12/trace/alert_TVD12.log
/u00/app/oracle/diag/rdbms/tvd12cdb/TVD12CDB/trace/alert_TVD12CDB.log

36. Some Samples
Analyzing Oracle related issues using TFA Collector36 09.09.2016
tfactl> access add –user ora01
Sucessfully added 'ora01' to TFA Access list.
.---------------------------------.
| TFA Users in training |
+-----------+-----------+---------+
| User Name | User Type | Status |
+-----------+-----------+---------+
| ora01 | USER | Allowed |
| oracle | USER | Allowed |
'-----------+-----------+---------'

37. Some Samples
Analyzing Oracle related issues using TFA Collector37 09.09.2016
tfactl> summary
Output from host : training
------------------------------
=====Nodes=====training=====Homes=====.
-------------------------------------------------------------------------------------------------------------.
| Home | Type | Version | Database | Instance | Patches |
+----------------------------------+------+------------+----------------+----------------+-------------------+
| /u00/app/oracle/product/12.1.0.2 | DB | 12.1.0.2.0 | TVD12CDB,TVD12 | TVD12CDB,TVD12 | 22139226,21948354 |
'----------------------------------+------+------------+----------------+----------------+-------------------'

38. Some Samples
Analyzing Oracle related issues using TFA Collector38 09.09.2016
tfactl> param block_size
tfactl> param block_size
Output from host : training
------------------------------
TVD12.training.trivadistraining.com.TVD12.TVD12.db_block_size = 8192
TVD12CDB.training.trivadistraining.com.TVD12CDB.TVD12CDB.db_block_size = 8192

39. Some Samples
Analyzing Oracle related issues using TFA Collector39 09.09.2016
tfactl> diagnosetfa -local
Running TFA Diagnostics...
Node List to collect TFA Diagnostics : 1 training
Running TFA Diagnostics on training...
Mon Aug 22 17:05:24 2016 : Collecting TFA Process details...
Mon Aug 22 17:05:24 2016 : Collecting Details of TFA Files...
... (TFA Install Logs, Disk Space, Top Output, TFA Status, JStack Output, TFA Logs,BDB Stats)
Mon Aug 22 17:05:29 2016 : Zipping Collections...
Sleeping for 10 Seconds...
TFA Diagnostics are being collected to /tmp/tfadiagnostics_20160822_170523 :
/tmp/tfadiagnostics_20160822_170523/training.zip

40. Analyzing Oracle related issues using TFA Collector40 09.09.2016
References

41. References
Markus Flechtner
– Principal Consultant Trivadis
– Tools nicht nur für den RAC (German slides)
– DOAG Regio NRW, 09.05.2016
– http://de.slideshare.net/markusdba/tools-nicht-nur-fr-den-rac
09.09.2016 Analyzing Oracle related issues using TFA Collector41

42. References
Oracle MOS ID 1513912.2
Trace File Analyzer Collector User Guide
09.09.2016 Analyzing Oracle related issues using TFA Collector42

43. Session Feedback – now
TechEvent September 201643 09.09.2016
Please use the Trivadis Events Mobile App to give session feedback
Use "My schedule" if you registered for this session
Otherwise use "Agenda" and the search function
If the mobile App does not work (or if you have a Windows Phone) use your Mobile
Browser
– URL: http://trivadis.quickmobileplatform.eu/
– Username: <your_loginname> (like svv)
– Password: sent by mail...

44. Rainer Klomps
Senior Consultant
Tel. +49 162 292 34 36
rainer.klomps@trivadis.com
09.09.2016 Analyzing Oracle related issues using TFA Collector51