NZOUG-GroundBreakers-2018 - Troubleshooting and Diagnosing 18c RAC

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Troubleshooting and Diagnosing 18c RAC
Sandesh Rao
VP Autonomous Health and Machine Learning
@sandeshr
https://www.linkedin.com/in/raosandesh/

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
2

Program Agenda
Architecture and Basics
Troubleshooting Scenarios
Proactive and Reactive tools
18/19c and beyond
Q&A
1
2
3
4
5
3

Program Agenda
18/19c and beyond
Q&A
1
2
3
4
5
4

Grid Infrastructure
• Grid Infrastructure is Combination of :
– Oracle Cluster Ready Services (CRS)
– Oracle Automatic Storage Management (ASM)
• The Grid Home contains the software for both
products
– Must be installed in different location to RDBMS home
– Installer locks the Grid Home path by setting root
permissions
• CRS can also be Standalone for ASM and/or Oracle
Restart
• CRS can run by itself or in combination with other
vendor clusterware
5
Overview
Disk Group A Disk Group B
Database
Instance
Database
Instance
ASM
Instance
ASM
Instance
Database
Instance
Database
Instance
ASM
Instance
ASM
Instance
Database
Instance
ASM
Instance
Host 1 Host 2 Host 3
Cluster
ASM Disk Groups

Grid Infrastructure
• Shared Oracle Cluster Registry (OCR) and Voting files
– Must be in ASM or CFS
– OCR backed up every 4 hours automatically GIHOME/cdata
– Kept 4,8,12 hours, 1 day, 1 week
– Restored with ocrconfig
– Voting file backed up into OCR at each change.
– Voting file restored with crsctl
CRS Requirements

Grid Infrastructure
• Requirements
– One or more redundant private networks for inter-node communications
– High speed with low latency
– Separate physical network or managed converged network
– VLANS are supported
• Usage
– Interconnect is a memory backplane for the cluster
– Clusterware messaging
– RDBMS messaging and block transfer
– ASM messaging
– HANFS for block traffic
CRS Network

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 8
CRS stack is spawned from
Oracle HA Services Daemon
(ohasd)
On Unix ohasd runs out of
inittab with respawn
A node can be evicted
when deemed unhealthy
• May require reboot
• IPMI integration
or diskmon in case of Exadata
CRS provides Cluster Time
Synchronization services
• Always runs but in observer
mode if ntpd configured
How it works
Grid Infrastructure

Core Resources
Grid Infrastructure Processes
HA Stack CRS Stack CRS Service
Level 0 Level 1 Level 2 Level 3 Level 4
INIT
ohasd
cssdmonitor
Network sources
SCAN VIP
Node VIP
ACFS Registry
GNS VIP
ASM Instance
Diskgroup
DB Resources
SCAN Listener
Listener
Services
eONS
ONS
GNS
GSD
CRSD
orarootagent
CRSD
oraagent
ASM
mDNSD
GIPCS
EVMD
GPNPD
CRSD
CTSSD
Diskmon
CSSD
OHASD
oraagent
OHASD
oraclerootagent
cssdagent

Oracle RAC 12c and onwards
Flex Cluster Flex ASM
Full Oracle
Multitenant & In-
Memory Support
Fleet
Provisioning and
Patching (FPP)
10
http://www.slideshare.net/MarkusMichalewicz/oracle-
database-inmemory-meets-oracle-rac
New In-Memory
Format
SALES
Column
Format
Oracle Confidential – Internal/Restricted/Highly Restricted

• Configure during Installation
• Reject non-Oracle I/O
• Stops OS utilities from
overwriting ASM disks
• Protects database files
• Reduce OS resource usage
• Fewer open file descriptors
• Faster node recovery
11
12.2 Automatic Storage Management (ASM)
ASM Filter Driver – Full Integration
• Further configuration and
monitoring is conducted by
using the AFDTOOL utility:
• Provision a disk:
$ afdtool -add /dev/dsk1 disk1
• Remove a disk:
$ afdtool -delete disk1
• List the managed disks:
$ afdtool -getdevlist
Oracle Confidential – Internal/Restricted/Highly Restricted

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted 12
Oracle RAC 12.2 Enhancements Worth Noticing
Node Weighting
Idea: If everything is
equal, let the majority of
work survive
Pluggable Database
& Service Isolation
Improved singleton
workload performance
and failure behavior
Service-oriented
Buffer Cache Access
Improved data access
performance & planned
maintenance operation
Fully Integrated
Extended RAC Support
Site-awareness and installer
support for extended RAC

Node Eviction Basics
13
Behavior pre-12.1.0.2
NodeA
Oracle GI | HUB
Oracle RAC
NodeB
Oracle GI | HUB
Oracle RAC
cons_1 cons_2
• Node eviction follows a
rather predictable pattern
– Example in a 2-node cluster: The node
with the lowest node number survives.
• Customers must not base their
application logic on which node
survives the split brain.
– As this may(!) change in future releases
✔

Node Weighting
14
Idea: Everything equal, let the majority of work survive
NodeA
Oracle GI | HUB
Oracle RAC
NodeB
Oracle GI | HUB
Oracle RAC
cons_1 cons_2
• Node Weighing is a new feature that considers
the workload run on a node during fencing
• The idea is to let the majority of work survive,
if everything else is equal
– “Majority work” is for example represented
by the number of services.
• Example: In a 2-node cluster, the node hosting
the majority of services (at fencing time) is
meant to survive
• DBAs can overrule and rate a service
as a “critical” based on business needs
✔

Node Weighting
work survive
Pluggable Database
& Service Isolation
Improved singleton
Service-oriented
Buffer Cache Access
Fully Integrated

Pluggable Database & Service Isolation
16
Prevents “noisy neighbors” from affecting others with unnecessary chatter
NodeA
Oracle GI | HUB
Oracle RAC
NodeB
Oracle GI | HUB
Oracle RAC
cons_1 cons_2
• Using Oracle Multitenant, PDBs can be opened
as singletons (in one database instance only), in
a subset of instances or all in instances at once.
• If certain PDBs are only opened on some
instances, Pluggable Database Isolation
– improves performance by
• Reducing DLM operations for
PDBs not open in all instances.
• Optimizing block operations based
on in-memory block separation.
MSG
Messages (MSG)

Pluggable Database & Service Isolation
17
Prevents instance failures of instances only hosting singleton PDBs to affect others
NodeA
Oracle GI | HUB
Oracle RAC
NodeB
Oracle GI | HUB
Oracle RAC
cons_1 cons_2
✔
• Using Oracle Multitenant, PDBs can be opened
as singletons (in one database instance only), in
a subset of instances or in all instances at once.
• If certain PDBs are only opened on some
instances, Pluggable Database Isolation
– Improves performance by
• Reducing DLM operations for
PDBs not open in all instances.
• Optimizing block operations based
on in-memory block separation.
– Ensures that instance failures of instances
only hosting singleton PDBs will not impact
other instances of the same RAC-based CDB.

Node Weighting
work survive
Pluggable Database
& Service Isolation
Improved singleton
Service-oriented
Buffer Cache Access
Fully Integrated

Service-oriented Buffer Cache Access
19
Improve performance by managing data with the service to which it belongs
NodeA
Oracle GI
Oracle RAC
NodeB
Oracle GI
Oracle RAC
cons_1 cons_2
• Service-oriented Buffer Cache Access over time
determines the data (on database object level)
accessed by the service. This information
– Is persisted in the database.
– Is used to improve data access performance
(e.g. do not manage data of a service in an
instance that does not host the service).
– Can be used to pre-warm an instance cache prior
to a service startup (fresh start or relocation).

Node Weighting
work survive
Pluggable Database
& Service Isolation
Improved singleton
Service-oriented
Buffer Cache Access
Fully Integrated

Stretch Cluster
DB1
+ASM1
NODE 1
DB2
+ASM2
NODE 2
DB3
+ASM3
NODE 3
DB4
NODE 4
SITE-A
Extended Cluster Architecture
+ASM4
SITE-B
SITE-C
FG1 FG2 FG3
QFG
FG4 FG5 FG6
SWITCH

Cluster Domain
22
For cost reduction through centralization, standardization and optimization
Why Use Oracle RAC for Your Private Database Cloud?
Cluster
Single
Node
Cluster Centralization:
centralize common management tasks
on the Domain Services Cluster.
Domain Services Cluster Standardization:
Use the same building blocks –
commodity hardware clusters – to
scale databases, compute & storage.
Database
Member
Cluster
Application
Member
Cluster
Optimization example:
Version independence – run any
Oracle RAC 12.2+ Member Cluster
using any platform at any time.
Linux
Cluster
AIX
Cluster
Solaris
Cluster

Centralization – Cluster Domain & Domain Services
Domain Services Cluster
Mgmt
Repository
Service
Trace File
Analyzer
(TFA)
Service
Rapid
Home
Provision
Service
Cluster Domain
A Cluster Domain is a logical management
entity to group various clusters in your DC.
The Mgmt Repository and the TFA service are
mandatory in the Cluster Domain. They represent
centralized versions of their local counterparts.
To provide centralized services in the Cluster
Domain, you need to deploy a Domain Services
Cluster. It will host the central services.
Additional services can
be added as needed.

Standardization – Member Clusters
Cluster Domain
Database
Member Cluster
uses
local ASM
Application
Member Cluster
GI only
A (Database) Member Cluster is a cluster that registers with the Mgmt Repository
Service and uses the centralized TFA service. It can use additional services as needed.
Mgmt
Repository
Service
Trace File
Analyzer
(TFA)
Service
Rapid
Home
Provision
Service
An Application Member Cluster (available
since 12.1.0.2) is a cluster designed to host
applications. It uses a lightweight GI stack.

Standardization – Storage Consolidation
Mgmt
Repository
Service
Trace File
Analyzer
(TFA)
Service
Rapid
Home
Provision
Service
Database
Member Cluster
uses the
ASM Services
Shared ASM
Cluster Domain
Storage Services
ASM
Service
IO
Service
ACFS
Services
Database
Member Cluster
uses the IO &
ASM Services
Storage flexibility: Member Clusters do not need
direct connectivity to shared disks. Using the
shared ASM Service, they can use network
connectivity to the IOservice to access a centrally
managed pool of storage.
To further standardize and centralize, various
Storage Services are offered in the domain.

Fleet Patching & Provisioning Support
Database & Grid Infrastructure
11.2.0.3.
11.2.0.4.
12.1
12.2
18
VM VM
VM VM
VM VM
VM VM
• Single Instance
• Oracle Restart
• Oracle RAC One
• Oracle RAC
BM
Non-CDB CDB/PDB
VM
• Generic Software
• Data Guard Aware
• Customizable
Multi-OS
19

Hides errors, timeouts, and
maintenance
No application knowledge or
changes to use
Rebuilds session state & in-flight
transactions
Adapts as applications change:
protected for the future
Standardize on Transparent Application Continuity
27
Request
Errors/Timeouts hidden
TAC
Applications see no errors during outages

Oracle RAC Performance Features
• Automatic Undo Management
• Cache Fusion
• Oracle Real Application Clusters
• Session Affinity
• PDB & Services Isolation
• Service-Oriented Buffer Cache
• Leaf Block Split Optimizations
• Self Tuning LMS
• Multithreaded Cache Fusion
• ExaFusion Direct-to-Wire Protocol
• Smart Fusion Block Transfer
• Universal Connection Pool (UCP) Support for Oracle RAC
• Support for Distributed Transactions (XA) in Oracle RAC
• Parallel Execution Optimizations for Oracle RAC
• Affinity Locking and Read-Mostly Objects
• Reader Bypass
• Flash Cache
• Connection Load Balancing
• Load Balancing Advisory
• Cluster Managed Services
• Automatic Storage Management
9i 10g
11g
12c
18c
• Scalable Sequences
• Undo RDMA-Read
• Commit Cache
• Database Reliability Framework
Over two decades of innovation

RAC Enhancements
• Remastering Slaves (1 slave per LMS)
– Starting with Oracle RAC 12.1, the LMS offloads heavy remastering work to the slave
– This improves LMS’s responsiveness for Cache Fusion requests during remastering
• Support for 100 LMS’s – change in default value
– Oracle RAC 12.2 supports up to 100 LMS’s (names: LMS0-LM99) as opposed to 35
– On larger systems (lots of CPU, large SGA), more LMS’s will start by default
– More LMS’s means better reconfiguration time without any impact during runtime
• More Dynamic Remastering (DRM)
– Starting with Oracle RAC 19c, DRM is planned to more adaptively consider the overall system state
29

Program Agenda
18/19c and beyond
Q&A
1
2
3
4
5
30

Cluster Startup
Oracle Support
TFA
Check core CRS
resources
running
ps –ef|grep init.ohasd
ps –ef|grep ohasd.bin
Not Running
Review status of
CRS services &
stack
crsctl check crs
crsctl check cluster
Running
Compare OLR
permissions to
reference
system & fix
differences
Not Running
Running
tfactl diagcollect
Review & fix
issues in logs
ohasd.log
Agent logs
process logs
Review & fix
CRS startup
config & log
crsctl config crs
ohasd.log

Node Eviction Problem Triage
Oracle Support
TFA
tfactl diagcollect
Check for &
fix
resource
starvation
System log
Troubleshooting guides:
1531223.1 (OSWatcher)
1328466.1 (CHM)
Check for &
fix
network
heartbeat
problems
ocssd.log
1050693.1
1534949.1
1546004.1
Check for &
fix
voting disk
problems
1549428.1
1466639.1

Reconfiguration Performance Improvements
11.2.0.4
11204
4 x
1.5 x
12.2 18.1

• Timings with different #LMS:
– Total reconfiguration time for an
instance leave & re-join
– 100GB cache
– 2 node RAC
34
Reconfiguration Performance as of 18c
Buffer Cache Size Reconfiguration Time
25GB 3.0 sec
50GB 4.9 sec
100GB 8.3 sec
• Timings with different cache sizes:
– Total reconfiguration time for an
instance leave & re-join
– 8 LMS’s
– 2 node RAC
# LMS Reconfiguration Time
8 LMS’s 8.3 sec
16 LMS’s 5.0 sec
32 LMS’s 3.6 sec

Reconfiguration Diagnosability
**************** BEGIN DLM RCFG HA STATS ****************
Total dlm rcfg time (inc 6): 3.586 secs (394926177, 394929763)
Begin step .........: 0.005 secs (394926177, 394926182)
Freeze step ........: 0.019 secs (394926182, 394926201)
Sync 1 step ........: 0.002 secs (394926264, 394926266)
Sync 2 step ........: 0.024 secs (394926266, 394926290)
Enqueue cleanup step: 0.002 secs (394926290, 394926292)
Sync pcm1 step .....: 0.004 secs (394926293, 394926297)
……
….
Enqueue dubious step: 0.004 secs (394926432, 394926436)
Sync 5 step ........: 0.000 secs (394926436, 394926436)
Enqueue grant step .: 0.001 secs (394926436, 394926437)
Sync 6 step ........: 0.012 secs (394926437, 394926449)
Fixwrt replay step .: 0.885 secs (394928837, 394929722)
Sync 8 step ........: 0.040 secs (394929722, 394929762)
End step ...........: 0.001 secs (394929762, 394929763)
Number of replayed enqueues sent / received .......: 2246 / 893
Number of replayed fusion locks sent / received ...: 124027 / 0
Number of enqueues mastered before / after rcfg ...: 2058 / 1384
**************** END DLM RCFG HA STATS *****************
Detailed timing
breakdown available
in LMON trace file

DRM Diagnosability
Dynamic Remastering Statistics DB/Inst: SALES/sales1 Snaps: 393-452
-> Affinity objects - Affinity objects mastered at the begin/end snapshot
-> Read-mostly objects - Read-mostly objects mastered at the begin/end snapshot
per Begin End
Name Total Remaster Op Snap Snap
-------------------------------- ------------ ------------- -------- --------
remaster ops 24 1.00
remastered objects 24 1.00
remaster time (s) 7.4 0.31
freeze time (s) 1.5 0.06
cleanup time (s) 2.4 0.10
replay time (s) 0.3 0.01
fixwrite time (s) 2.4 0.10
sync time (s) 0.1 0.01
affinity objects N/A 3 27
read-mostly objects N/A 0 0
read-mostly objects (persistent) N/A 0 0
Detailed timing
breakdown available
in AWR Report

Program Agenda
18/19c and beyond
Q&A
1
2
3
4
5
37

Oracle’s Database and Clusterware Tools
• What if issues were detected before they
had an impact?
• What if you were notified with a specific
diagnosis and corrective actions?
• What if resource bottlenecks threatening
SLAs were identified early?
• What if bottlenecks could be
automatically relieved just in time?
• What if database hangs and node
reboots could be eliminated?
Confidential – Oracle Restricted 38
Cluster
Verification
Utility
ORAchk /
EXAchk
Cluster
Health
Monitor
Cluster
Health
Advisor
Trace File
Analyzer
Hang
Manager
Memory
Guard
Quality of
Service
Management

Automatic proactive warning
of problems before they
impact you
39
Get scheduled health reports
sent to you in email
Why Oracle ORAchk & EXAchk
Health checks for most impactful
reoccurring problems
Runs in your environment
with no need to send
anything to Oracle
Findings can be integrated
into other tools of choiceEngineered
Systems
Non
Engineered
Systems
EXAchk
Common Framework
ORAchk
Further slide details

If Oracle Clusterware is installed
1. Download latest version
– ORAchk : 1268927.2
– EXAchk : 1070954.1
2. Copy the zip file to the installation
system and extract:
– EXAchk:
in /opt/oracle.SupportTools/exachk as the
Oracle Grid Infrastructure home owner
– ORAchk:
in CRS_HOME/suptools/orachk as the
Oracle Grid Infrastructure home owner
If Oracle Clusterware is not installed
1. Download latest version
– ORAchk : 1268927.2
– EXAchk : 1070954.1
2. Copy the zip file to the installation
system and extract:
– EXAchk:
in /opt/oracle.SupportTools/exachk as the
root user
– ORAchk:
in a convenient location as Oracle software
install user or Oracle Database home owner
40
Installation

Engineered Systems
Oracle Exadata Database Machine
Oracle SuperCluster
Oracle Private Cloud Appliance
Oracle Database Appliance
Oracle Big Data Appliance
Oracle Exalogic Elastic Cloud
Oracle Exalytics In-Memory
Machine
Oracle Zero Data Loss Recovery
Appliance
Oracle ZFS Storage Appliance
Systems
Oracle Solaris
Cross stack checks
Solaris Cluster
OVN
ASR
41
Oracle Stack Coverage
Oracle Database
Standalone Database
Grid Infrastructure & RAC
Maximum Availability Architecture
(MAA) Scorecard
Upgrade Readiness Validation
Golden Gate
Enterprise Manager Cloud
Control
Repository
Agent
OMS
Middleware
Application Continuity
Oracle Identify and Access
Management Suite (Oracle IAM)
E-Business Suite
Oracle Payables
Oracle Workflow
Oracle Purchasing
Oracle Order Management
Oracle Process Manufacturing
Oracle Receivables
Oracle Fixed Assets
Oracle HCM
Oracle CRM
Oracle Project Billing
Siebel
Database best practices
PeopleSoft
Database best practices
SAP
EXAdata best practices

ORAchk
Operating System Versions Supported
Linux / zLinux Oracle Linux/RedHat 4,
5, 6, 7
SuSE 9,10, 11, 12
Oracle Solaris SPARC / x86-64 9, 10, 11
AIX 5.2, 5.3, 6.1, 7.1, 7.2
HPUX Itanium / PA-RISC 11.23, 11.31
Microsoft Windows 2008, 2012
EXAchk
• All Oracle Engineered Systems
– Apart from ODA, which uses ORAchk
42
Supported Environments

• Run as root (recommended)
o ORAchk/EXAchk will su to lower privileged
owners of RDBMS or grid homes
o To specify a user other that root for these
situations:
• Run as RDBMS or GRID Home Owner
o User must be able to switch to root for root level
checks – several options:
1. Provide the root userid password at prompts
or
2. Set up sudo
or
3. Pre-configure passwordless SSH connectivity
Which User to Run as
Connect via
SSH &
Run Checks on
Default User
Change User By
exporting user id in this
Environment Variable
Exadata Storage
Server
root RAT_CELL_SSH_USER
InfiniBand
switches
root
(when run as root)
RAT_IBSWITCH_USERnm2user
(when run as other user)
43
Note: You may only choose from the provided lower privileged account
Note:
•On SuperCluster you can use Role Based Access Control (RBAC) to
execute root privileged checks, no root user required.
•root checks must be run as a user with a root equivalent access role
•On Exalogic it is only supported to run as root

• Profiles provide logical grouping of
checks which are about similar topics
• Run only checks in a specific profile
• Run everything except checks in a specific
profile
Profiles
./exachk –profile <profile>
./exachk –excludeprofile <profile>
Profile Description
asm ASM Checks
avdf Audit Vault Configuration checks
clusterware Oracle clusterware checks
control_VM Checks only for Control VM(ec1-vm, ovmm, db, pc1, pc2).
No cross node checks
corroborate Exadata checks needs further review by user to determine
pass or fail
dba DBA Checks
ebs Oracle E-Business Suite checks
eci_healthchecks Enterprise Cloud Infrastructure Healthchecks
ecs_healthchecks Enterprise Cloud System Healthchecks
goldengate Oracle GoldenGate checks
hardware Hardware specific checks for Oracle Engineered systems
maa Maximum Availability Architecture Checks
ovn Oracle Virtual Networking
platinum Platinum certification checks
preinstall Pre-installation checks
prepatch Checks to execute before patching
security Security checks
solaris_cluster Solaris Cluster Checks
storage Oracle Storage Server Checks
switch Infiniband switch checks
sysadmin Sysadmin checks
user_defined_checks Run user defined checks from user_defined_checks.xml
44

• Profiles provide logical grouping of
checks which are about similar topics
• Run only checks in a specific profile
• Run everything except checks in a specific
profile
Profiles
./orachk –profile <profile>
./orachk –excludeprofile <profile>
Profile Description
asm ASM Checks
bi_middleware Oracle Business Intelligence checks
clusterware Oracle clusterware checks
dba DBA Checks
ebs Oracle E-Business Suite checks
emagent Cloud control agent checks
emoms Cloud Control management server
em Cloud control checks
goldengate Oracle GoldenGate checks
hardware Hardware specific checks for Oracle Engineered systems
oam Oracle Access Manager checks
oim Oracle Identify Manager checks
oud Oracle Unified Directory server checks
ovn Oracle Virtual Networking
peoplesoft Peoplesoft best practices
preinstall Pre-installation checks
prepatch Checks to execute before patching
security Security checks
siebel Siebel Checks
solaris_cluster Solaris Cluster Checks
storage Oracle Storage Server Checks
switch Infiniband switch checks
sysadmin Sysadmin checks
user_defined_checks Run user defined checks from user_defined_checks.xml
45

Enterprise Manager Integration
•Check results integrated into EM
compliance framework via plugin
•View results in native EM
compliance dashboards
•Related checks grouped into
compliance standards
•View targets checked, violations &
average score
•Drill down into compliance standard
to see individual check results
•View break down by target
46

• Integration is via the Enterprise
Manager ORAchk Healthchecks
plugin with the following Support:
• The following prerequisites must be met
before you can deploy the plug-in:
o Verify that your Engineered Systems hardware
and software are at the supported level as
described in Supported Hardware and Software
Versions
o All Engineered System plug-ins should be
deployed
o InfiniBand switches and storage cells should be
an Enterprise Manager-managed target for the
respective engineered system
o Expect package should be installed on the
hosts
Enterprise Manager Plugin Prerequisites
Hardware Types
Supported
By Plugin
Exadata (physical configuration only) Yes
Exadata (virtual configuration) No
Recovery appliance Yes
Exalogic (physical configuration) Yes
Exalogic (virtualized configuration) Yes
Oracle SuperCluster No
Oracle Private Cloud Machine No
47

JSON Output to Integrate with Kibana, Elastic Search etc
48

Oracle Health Check Collection Manager Dashboard
49

Differences between each run
Diff Output
50

• New checks to help when upgrading the database
to 12.2
• Both pre and post upgrade verification to prevent
problems related to:
• OS configuration
• Grid Infrastructure & Database patch prerequisites
• Database configuration
• Cluster configuration
Upgrade to Database 12.2 with confidence
orachk -u –o pre
orachk -u –o post
Pre upgrade
Post upgrade

Automatically run on Exadata user domain (DomU)
• EXAchk will be automatically setup on Exadata user domain (DomU) as part of TFA
install
• Autostart will configure the daemon to restart at 1am every morning to rediscover any
environment changes
• Full Local client run will be triggered at 2am every morning
• Most impactful checks will be run every 2 hours via the exatier1 profile
• Any collections older than 2 weeks will be automatically purged
• To configure Autostart from exachk standalone install use:
• Once enabled daemon settings can be changed as per normal
• Remove with: or
exachk -autostart
exachk -autostop tfactl run exachk -autostop

• Health Check information is constantly changing
over time
– What is applicable today, may not be applicable several
months later
– This is the reason ORAchk & EXAchk have always had a
quarterly release cycle
• The older Health checks are the more risk of them
containing out of date information
• If ORAchk/EXAchk is more than one year old, it will
not execute any checks
1 Year
Lifetime
Confidential – Oracle Internal/Restricted/Highly Restricted 53
Ensure only up to date checks are run

• Many new checks focus on known issues in 12c
Optimizer as well as SQL Plan Management
• These checks target problems such as:
– Wrong results returned
– High memory & CPU usage
– Errors such as ORA-00600 or ORA-07445
– Issues with cursor usage
– Other general SQL plan management problems
All contained in the dba profile:
Improve performance of SQL queries
orachk -profile dba

300%
Faster
Completely rewritten, with a focus on
performance and extensibility
Internal testing results show this
version runs up to 300% faster than
12.2.0.1.3.
Performance is particularly improved
on lower resourced environments
Confidential – Oracle Internal/Restricted/Highly Restricted 55

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal 56
Option to Only Run Checks That Previously Failed
Generate a health check
report and identify
problems
Find Problems Fix Verify
Fix the issues identified Generate another health
check report, verifying
only the issues that
failed before are fixed
-failedchecks <previous_result>

$ ./orachk -fileattr start
CRS stack is running and CRS_HOME is not set. Do you want to set CRS_HOME to
/u01/app/11.2.0.4/grid?[y/n][y]
Checking ssh user equivalency settings on all nodes in cluster
Node mysrv22 is configured for ssh user equivalency for oradb user
Node mysrv23 is configured for ssh user equivalency for oradb user
List of directories(recursive) for checking file attributes:
/u01/app/oradb/product/11.2.0/dbhome_11203
/u01/app/oradb/product/11.2.0/dbhome_11204
orachk has taken snapshot of file attributes for above directories at:
/orahome/oradb/orachk/orachk_mysrv21_20170504_041214
• Track changes to the attributes of important files with –fileattr
– Looks at all files & directories within Grid Infrastructure and Database homes by default
– The list of monitored directories and their contents can be configured to your specific requirements
– Use –fileattr start to start the first snapshot
57
Keep Track of Changes to the Attributes of Important Files
./orachk –fileattr start

Note:
• Use the same arguments with check that you used with start
• Will proceed to perform standard health checks after attribute checking
• File Attribute Changes will also show in HTML report output
$ ./orachk -fileattr check -includedir "/root/myapp/config" -excludediscovery
CRS stack is running and CRS_HOME is not set. Do you want to set CRS_HOME to
/u01/app/18/19c.0/grid?[y/n][y]
Checking for prompts on myserver18 for oragrid user...
Checking ssh user equivalency settings on all nodes in cluster
Node myserver17 is configured for ssh user equivalency for root user
List of directories(recursive) for checking file attributes:
/root/myapp/config
Checking file attribute changes...
.
"/root/myapp/config/myappconfig.xml" is different:
Baseline : 0644 oracle root /root/myapp/config/myappconfig.xml
Current : 0644 root root /root/myapp/config/myappconfig.xml
…etc
…etc
• Compare current attributes against first snapshot using –fileattr check
58
Keep Track of Changes to the Attributes of Important Files
./orachk –fileattr check
• Results of snapshot comparison will also
be shown in the HTML report output

• OraChk can help compare and
contrast different configurations
– Different I/O scheduler?
– Centralized location makes it easier to
identify diffs
• Daemon Mode
– $orachk –d start/stop/status
• Configure Schedule
– $orachk AUTORUN_SCHEDULE
• What’s difference between OraChk
and ExaChk?
Benefit from a Centralized OraChk
11204
11204DBA 1
DBA 2
Oracle RAC Assessment Report
System Health Score is 89 out of 100
Oracle RAC Assessment Report
System Health Score is 98 out of 100

Easy to use real-time
health monitoring,
fault detection & diagnosis
via a single interface
Secure consolidation of
distributed diagnostic
collections
Continuous availability so
you always get what’s
needed for resolution
Machine learning driven,
autonomous degradation
detection, reduces your
overheads
Why TFA?

Autonomous Usage
Oracle Grid Infrastructure
& Databases
Oracle Support
TFA
1
TFA detects a fault
2Diagnostics
are collected
3
Distributed diagnostics
are consolidated and
packaged
4
Notification of fault is sent
5 Diagnostic collection is
uploaded to Oracle
Support for root cause
analysis & resolution

Real-time Status Summary
tfactl summary
Choose an
option to drill
down
High-level summary of all
Database components

Real-time Status Summary – Drill Down
Drill downs show real-time
analytics & details of any
problems found

• Analyze all important recent log entries: • Search recent log entries:
64
Investigate Logs & Look for Errors
tfactl analyze –last 1d tfactl analyze -search “ora-00600" -last 8h
Searching for
“ora-00600”

Perform Analysis Using the Included Tools
Not all tools are included in Grid or Database install.
Download from 1513912.1 to get full collection of tools
Tool Description
orachk or
exachk
Provides health checks for the Oracle stack.
Oracle Trace File Analyzer will install either
• Oracle EXAchk for Engineered Systems, see document 1070954.1 for
more details
or
• Oracle ORAchk for all non-Engineered Systems, see document
1268927.2 for more details
oswatcher Collects and archives OS metrics. These are useful for instance or node
evictions & performance Issues. See document 301137.1 for more details
procwatcher Automates & captures database performance diagnostics and session level
hang information. See document 459694.1 for more details
oratop Provides near real-time database monitoring. See document 1500864.1
for more details.
alertsummary Provides summary of events for one or more database or ASM alert files
from all nodes
ls Lists all files TFA knows about for a given file name pattern across all nodes
pstack Generate process stack for specified processes across all nodes
Tool Description
grep Search alert or trace files with a given database and file name pattern, for
a search string.
summary Provides high level summary of the configuration
vi Opens alert or trace files for viewing a given database and file name
pattern in the vi editor
tail Runs a tail on an alert or trace files for a given database and file name
pattern
param Shows all database and OS parameters that match a specified pattern
dbglevel Sets and unsets multiple CRS trace levels with one command
history Shows the shell history for the tfactl shell
changes Reports changes in the system setup over a given time period. This
includes database parameters, OS parameters and patches applied
calog Reports major events from the Cluster Event log
events Reports warnings and errors seen in the logs
managelogs Shows disk space usage and purges ADR log and trace files
ps Finds processes
triage Summarize oswatcher/exawatcher data
65
Verify which tools you have installed: tfactl toolstatus

Collecting Diagnostics with TFA
Standard Diag Collection Targeted Diag Collection via SRDC
1. Run
2. Upload resulting zip file to SR
tfactl diagcollect –srdc <srdc>1. Run
OR
Run
OR
Run
tfactl diagcollect –last <n><d>|<h>
tfactl diagcollect –from <date> -to <time>
tfactl diagcollect
66

• For certain types of problems
Oracle Support will ask you to
run a Service Request Data
Collection (SRDC)
• Previously this would have
involved:
• Reading many different
support documents
• Collecting output from
many different tasks
• Gathering lots of different
diagnostics
• Packaging & uploading
• Now just run:
One Command SRDCs
tfactl diagcollect -srdc <srdc_type> -sr <SR#>
67

Manual Method
1. Generate ADDM reviewing Document 1680075.1 (multiple steps)
2. Identify “good” and “problem” periods and gather AWR reviewing
Document 1903158.1 (multiple steps)
3. Generate AWR compare report (awrddrpt.sql) using “good” and
“problem” periods
4. Generate ASH report for “good” and “problem” periods reviewing
5. Collect OSWatcher data reviewing Document 301137.1 (multiple
steps)
6. Collect Hang Analyze output at Level 4
7. Generate SQL Healthcheck for problem SQL id using Document
1366133.1 (multiple steps)
8. Run support provided sql scripts – Log File sync diagnostic output using
9. Check alert.log if there are any errors during the “problem” period
10. Find any trace files generated during the “problem” period
11. Collate and upload all the above files/outputs to SR
Automated One Command TFA SRDC
1. Run
68
Targeted Diagnostics – Service Request Data Collections (SRDCs)
tfactl diagcollect –srdc dbperf
[-sr <sr_number>]
Confidential – Oracle Internal

New SRDCs
69
Type of Problem SRDC
General ORA Errors
• ORA-01031
• ORA-01578
• ORA-08102
• ORA-08103
Problems showing alert log messages of "Corrupt block relative dba" • dbblockcorruption
ASM/DBFS/DNFS/ACFS problems • dbfs
Create/maintain partitioned/subpartitioned table/index problems • dbpartition
Alow create/alter/drop commands against partitioned table/index • dbpartitionperf
SQL performance problems • dbsqlperf
UNDO corruption problems • dbundocorruption
Exalogic full Exalogs data collection information • esexalogic
Listener errors : TNS-12516 / TNS-12518 / TNS-12519 / TNS-12520 • listener_services
Naming service errors: ORA-12154 / ORA-12514 / ORA-12528 • naming_services
Standard Information for Database Auditing • dbaudit
tfactl diagcollect –srdc srdc_name

Collect ORA-00600 SRDC
bash-4.1$ ./tfactl diagcollect -srdc ORA-00600
Enter the time of the ORA-00600 [YYYY-MM-DD HH24:MI:SS,=ALL] :
Enter the Database Name [=ALL] :
1. Oct/18/2018 02:38:37 : [ogg11204] ORA-00600: internal error code, arguments: [ktfbtgex-7], [1015817],
[1024], [1015816], [], [], [], [], [], [], [], []
2. Oct/18/2018 02:38:25 : [ogg11204] ORA-00600: internal error code, arguments: [ksprcvsp2],
[1596993584], [], [], [], [], [], [], [], [], [], []
Please choose the event : 1-2 [1]
70

Selected value is : 1 ( Oct/18/2018 02:38:37 )
Scripts to be run by this srdc: ipspack rdahcve1210 rdahcve1120 rdahcve1110
Components included in this srdc: OS CRS DATABASE
Collecting data for local node(s)
Scanning files from Oct/17/2018 20:38:37 to Oct/18/2018 08:38:37
WARNING: End time entered is after the current system time.
Collection Id : 20181018032231myserver69
Detailed Logging at :
/scratch/app/oragrid/tfa/repository/srdc_ora600_collection_Thu_Oct_18_03_22_31_PDT_2018_node_loca
l/diagcollect_20181018032231_myserver69.log
2018/10/18 03:22:36 PDT : NOTE : Any file or directory name containing the string .com will be renamed to
replace .com with dotcom
71

.-----------------------------------------------------.
| Collection Summary |
+----------------+---------------+---------+-------+
| Host | Status | Size | Time |
+----------------+---------------+--------+--------+
| myserver69| Completed| 2MB | 97s |
'-----------------+---------------+--------+---------'
Logs are being collected to:
/scratch/app/oragrid/tfa/repository/srdc_ora600_collection_Thu_Oct_18_03_22_31_PDT_2018_node_local
/scratch/app/oragrid/tfa/repository/srdc_ora600_collection_Thu_Oct_18_03_22_31_PDT_2018_node_local
/myserver69.tfa_srdc_ora600_Thu_Oct_18_03_22_31_PDT_2018.zip
72

Use ‘tfactl’ to check for recent Errors
bash-4.1# tfactl events
Output from host : myserver69
INFO :2
ERROR :2
WARNING :0
Event Timeline:
[Oct/18/2018 02:38:25.000]: [db.ogg11204.ogg112041]: Incident details in:
/scratch/app/oradb/diag/rdbms/ogg11204/ogg112041/incident/incdir_102702/ogg112041_ora_5001_i102
702.trc
[Oct/18/2018 02:38:25.000]: [db.ogg11204.ogg112041]: ORA-00600: internal error code, arguments:
[ksprcvsp2], [1596993584], [], [], [], [], [], [], [], [], [], []
[Oct/18/2018 02:38:37.000]: [db.ogg11204.ogg112041]: Incident details in:
/scratch/app/oradb/diag/rdbms/ogg11204/ogg112041/incident/incdir_102703/ogg112041_ora_5001_i102
703.trc
[Oct/18/2018 02:38:37.000]: [db.ogg11204.ogg112041]: ORA-00600: internal error code, arguments:
[ktfbtgex-7], [1015817], [1024], [1015816], [], [], [], [], [], [], [], []
73

Check to see if a change may have caused the issue ?
-bash-4.1# tfactl changes
Output from host : myserver69
------------------------------
[Oct/17/2018 04:54:15.397]: [RDBMS.myDB1]: Parameter: parallel_max_servers: Value: 8 => 16
[Oct/17/2018 05:12:13.344]: [RDBMS.myDB1]: Parameter: log_archive_dest_1: Value: /var => /opt
74

Metadata search capability
• All metadata stored in the TFA index is searchable:
• Searching for all events for a database between certain dates:
75
tfactl search -showdatatypes|-json [json_details]
tfactl search -json
‘{
"data_type":"event",
"content":"oracle",
"database":"rac11g",
"from":“10/01/2018 00:00:00",
"to":"10/21/2018 00:00:00"
}’

• Listing all index events:
• Listing all available datatypes:
76
tfactl search -json ‘{"data_type":"event"}’
tfactl search -showdatatypes

REST service
• REST support allows invocation & query over HTTPS
• Oracle REST Data Services (ORDS) is included within the install
• Once ORDS is running REST invocations can be made via:
• Post to Result
• Can be downloaded with:
77
tfactl rest [-status|-start|-stop|-uninstall] [-dir ] [-port ] [-user ] [-debug [-level ]]
https://host:port/ords/api
https://myhost:9090/ords/tfactl/diagcollect {
"collectionId" : "20180111011121slc12ekf",
"zipName" : "TFA_DEF_ZIP_20180111011121",
"tagName" : "TFA_DEF_TAG_20180111011121"
}
https://myhost:9090/ords/tfactl/download/20180111011121slc12ekf
The tfactl rest command
can only be run by root
user
The standalone ORDS setup feature utilizes file based user authentication and is provided solely for use in test and development environments.
For production use, the included ords.war should be deployed and configured.

• All metadata stored in the TFA index is searchable using:
• Searching for all events for a database between certain dates can be done like this:
• Listing all index events can be done with:
• Listing all available datatypes can be done with :
78
tfactl search -showdatatypes|-json [json_details]
tfactl search -json
"{
"data_type":"event",
"content":"oracle",
"database":"rac11g",
"from":"01/20/2017 00:00:00",
"to":"12/20/2018 00:00:00"
}"
tfactl search -json "{"data_type":"event"}"
tfactl search -showdatatypes

OS Watcher (Support Tools Bundle)
Collect & Archive OS Metrics
• Executes standard UNIX utilities (e.g. vmstat, iostat, ps,
etc) on regular intervals
• Built in Analyzer functionality to summarize, graph and
report upon collected metrics
• Output is Required for node reboot and performance
issues
• Simple to install, extremely lightweight
• Runs on ALL platforms (Except Windows)
• MOS Note: 301137.1 – OS Watcher Users Guide

Procwatcher (Support Tools Bundle)
Monitor & Examine Database Processes
• Single instance & RAC
• Generates session wait, lock and latch reports as well as call stacks
from any problem process(s)
• Ability to collect stack traces of specific processes using Oracle Tools
and OS Debuggers
• Typically reduces SR resolution for performance related issues
• Runs on ALL major UNIX Platforms
• MOS Note: 459694.1 – Procwatcher Install Guide

oratop (Support Tools Bundle)
Near Real-Time Database Monitoring
• Single instance & RAC
• Monitoring current database activities
• Database performance
• Identifying contentions and bottleneck

Automatic Database Log Purge
• TFA can automatically purge database logs
– OFF by default
– Except on a Domain Service Cluster (DSC),
which it is ON by default
• Turn auto purging on or off:
• Will remove logs older than 30 days
– configurable with:
• Purging runs every 60 minutes
– configurable with:
82
tfactl set manageLogsAutoPurge=<ON|OFF>
tfactl set manageLogsAutoPurgePolicyAge=<n><d|h>
tfactl set manageLogsAutoPurgeInterval=<minutes>

Manual Database Log Purge
• TFA can manage ADR log and trace files
– Show disk space usage of individual diagnostic destinations
– Purge these file types based on diagnostic location and or age:
• "ALERT“, "INCIDENT“, "TRACE“, "CDUMP“, "HM“, "UTSCDMP“, "LOG“
tfactl managelogs <options>
Runs as the ADR home
owner. So will only be able
to purge files this owner
has permission to delete
Option Description
–show usage Shows disk space usage per diagnostic directory for both GI and database logs
-show variation –older <n><m|h|d> Use to determine per directory disk space growth.
Shows the disk usage variation for the specified period per directory.
-purge –older <n><m|h|d> Remove all ADR files under the GI_BASE directory, which are older than the time specified
–gi Restrict command to only diagnostic files under the GI_BASE
–database [all | dbname] Restrict command to only diagnostic files under the database directory. Defaults to all,
alternatively specify a database name
-dryrun Use with –purge to estimate how many files will be affected and how much disk space will be
freed by a potential purge command.
May take a while for a
large number of files
83

tfactl managelogs –show usage tfactl managelogs –show variation –older <n><m|h|d>
Use -gi to only
show grid
infrastructure
Use –database to only
show database

tfactl managelogs –purge –older n<m|h|d> -dryrun tfactl managelogs –purge –older n<m|h|d>
Use –dryrun
for a “what if”

Collection Uploaded to My Oracle Support
At the end of the collection the file is automatically
uploaded to the SR

With Wallet: Without wallet:
87
Upload any File
tfactl upload -wallet -sr <sr_number> <file1 file2> tfactl upload -user <user_id> -sr <sr_number> <file1 file2>

Faster & Easier SR Data Collection
tfactl diagcollect –srdc <srdc_type> -sr <SR#>
ORA Errors
• ORA-00020
• ORA-00060
• ORA-00600
• ORA-00700
• ORA-01555
• ORA-01628
• ORA-04030
• ORA-04031
• ORA-07445
• ORA-27300
• ORA-27301
• ORA-27302
• ORA-30036
Other internal database errors • internalerror
Database performance • dbperf
Database patching
• dbpatchinstall
• dbpatchconflict
Database resource • dbunixresources
XDB installation or invalid object • dbxdb
Database install / upgrade
• dbinstall
• dbupgrade
• dbpreupgrade
Database storage • asm
Excessive SYSAUX Space used by the Automatic
Workload Repository (AWR)
• dbawrspace
Database startup / shutdown • dbshutdown
• dbstartup
Data Guard • dbdataguard
Enterprise Manager tablespace usage metric • emtbsmetrics
Enterprise Manager general metrics page or
threshold problems - Run all three SRDCs
• emdebugon
• emdebugoff
• emmetricalert
Enterprise Manager target discovery / add
• emcliadd
• emclusdisc
• emdbsys
• emgendisc
• emprocdisc
Enterprise Manager OMS restart • emrestartoms

Manual Data Gathering vs One Command SRDC
Manual Data Gathering
1. Generate ADDM reviewing Document 1680075.1
2. Identify “good” and “problem” periods and gather AWR
reviewing Document 1903158.1
3. Generate AWR compare report (awrddrpt.sql) using “good”
and “problem” periods
4. Generate ASH report for “good” and “problem” periods
reviewing Document 1903145.1
5. Collect OSWatcher data reviewing Document 301137.1
6. Check alert.log if there are any errors during the “problem”
period
7. Find any trace files generated during the “problem” period
8. Collate and upload all the above files/outputs to SR
TFA SRDC
1. Run
tfactl diagcollect –srdc dbperf
89

1. Enter default for event date/time and database name
2. Scans system to identify recent 10 events in the system (ORA600
example shown)
3. Once the relevant event is chosen, proceeds with diagnostic
collection
90
One Command SRDC
tfactl diagcollect –srdc <srdc_type>
4. All required files are
identified
5. Trimmed where
applicable
6. Package in a zip ready
to provide to support
Interactive Mode

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | Confidential – Oracle Internal 91
Detecting degradations and faults
TFA DB Utils
• Health score based on
recent unresolved
warnings or errors in
database components
• Availability score
based on amount of
database components
up and running
• Capacity score shows
future capacity of all
database component
• Performance score
based on recent
database performance
TFA
DB
Utils
HealthAvailability
CapacityPerformance

Generates Diagnostic Metrics View of Cluster and Databases
Cluster Health Monitor (CHM)
GIMR
ologgerd
(master)
osysmond
osysmond
osysmond
osysmond
12c Grid Infrastructure
Management Repository
• Always on - Enabled by default
• Provides Detailed OS Resource Metrics
• Assists Node eviction analysis
• Locally logs all process data
• User can define pinned processes
• Listens to CSS and GIPC events
• Categorizes processes by type
• Supports plug-in collectors (ex.
traceroute, netstat, ping, etc.)
• New CSV output for ease of analysis
OS Data OS Data
OS Data
OS Data
Confidential – Oracle Internal/Restricted/Highly RestrictedConfidential – Oracle Restricted

Oclumon CLI or Full Integration with EM Cloud Control
Confidential – Oracle Internal/Restricted/Highly RestrictedConfidential – Oracle Restricted

Cluster Health Advisor (CHA)*
Discovers Potential Cluster & DB Problems - Notifies with Corrective Actions
94
OS Data
GIMR
ochad
• Detects node and database
performance problems
• Provides early-warning alerts and
corrective action
• Supports on-site calibration to improve
sensitivity
• Integrated into EMCC Incident Manager
and notifications
• Standalone Interactive GUI Tool
DB Data
CHM
Node
Health
Prognostics
Engine
Database
Health
Prognostics
Engine
* Requires and Included with RAC or R1N License
Confidential – Oracle Restricted

Calibrating CHA to your RAC deployment
Choosing a Data Set for Calibration – Defining “normal”
$ chactl query calibration –cluster –timeranges ‘start=2016-10-28 07:00:00,end=2016-10-28 13:00:00’
Cluster name : mycluster
Start time : 2016-10-28 07:00:00
End time : 2016-10-28 13:00:00
Total Samples : 11524
Percentage of filtered data : 100%
1) Disk read (ASM) (Mbyte/sec)
MEAN MEDIAN STDDEV MIN MAX
0.11 0.00 2.62 0.00 114.66
<25 <50 <75 <100 >=100
99.87% 0.08% 0.00% 0.02% 0.03%
2) Disk write (ASM) (Mbyte/sec)
0.01 0.00 0.15 0.00 6.77
<50 <100 <150 <200 >=200
100.00% 0.00% 0.00% 0.00% 0.00%
3) Disk throughput (ASM) (IO/sec)
2.20 0.00 31.17 0.00 1100.00
<5000 <10000 <15000 <20000 >=20000
100.00% 0.00% 0.00% 0.00% 0.00%
4) CPU utilization (total) (%)
9.62 9.30 7.95 1.80 77.90
<20 <40 <60 <80 >=80
92.67% 6.17% 1.11% 0.05% 0.00%

Calibrating CHA to your RAC deployment
• Create and store the new model
$ chactl query calibrate cluster –model daytime –timeranges ‘start=2018-10-28 07:00:00,
end=2018-10-28 13:00:00’
• Begin using the new model
$ chactl monitor cluster –model daytime
• Confirm the new model is being used
$ chactl status –verbose
monitoring nodes svr01, svr02 using model daytime
monitoring database qoltpacdb, instances oltpacdb_1, oltpacdb_2 using model DEFAULT_DB
Creating a new CHA Model with CHACTL

Cluster Health Advisor – Command Line Operations
Monitoring Your Databases and Nodes with CHACTL
Enable CHA monitoring on RAC database with optional model
$ chactl monitor database –db oltpacdb [-model model_name]
Enable CHA monitoring on RAC database with optional verbose
$ chactl status –verbose
monitoring nodes svr01, svr02 using model DEFAULT_CLUSTER
monitoring database oltpacdb, instances oltpacdb_1, oltpacdb_2 using model DEFAULT_DB

CHA Command Line Operations
Checking for Health Issues and Corrective Actions with CHACTL QUERY DIAGNOSIS
$ chactl query diagnosis -db oltpacdb -start "2016-10-28 01:52:50" -end "2016-10-28 03:19:15"
2016-10-28 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_1) [detected]
2016-10-28 01:47:10.0 Database oltpacdb DB Control File IO Performance (oltpacdb_2) [detected]
2016-10-28 02:59:35.0 Database oltpacdb DB Log File Switch (oltpacdb_1) [detected]
2016-10-28 02:59:45.0 Database oltpacdb DB Log File Switch (oltpacdb_2) [detected]
Problem: DB Control File IO Performance
Description: CHA has detected that reads or writes to the control files are slower than expected.
Cause: The Cluster Health Advisor (CHA) detected that reads or writes to the control files were
slow because of an increase in disk IO.
The slow control file reads and writes may have an impact on checkpoint and Log Writer (LGWR) performance.
Action: Separate the control files from other database files and move them to faster disks or Solid
State Devices.
Problem: DB Log File Switch
Description: CHA detected that database sessions are waiting longer than expected
for log switch completions.
Cause: The Cluster Health Advisor (CHA) detected high contention during log switches
because the redo log files were small and the redo logs switched frequently.
Action: Increase the size of the redo logs.

Cluster Health Advisor – Command Line Operations
HTML Diagnostic Health Output Available (-html <file_name>)

Oracle 12c Hang Manager
• Reliably detects database hangs and
deadlocks
• Autonomously resolves them
• Supports QoS Performance Classes, Ranks
and Policies to maintain SLAs
• Logs all detections and resolutions
• New SQL interface to configure sensitivity
(Normal/High) and trace file sizes
Autonomously Preserves Database Availability and Performance Session
DIA0
EVALUATE
DETECT
ANALYZE
Hung?
VERIFY
Victim
QoS
Policy

Full Resolution Dump Trace File and DB Alert Log Audit Reports
Oracle 12c Hang Manager
Dump file …/diag/rdbms/hm6/hm62/incident/incdir_5753/hm62_dia0_12656_i5753.trc
Oracle Database 12c Enterprise Edition Release 18/19c.0.0.0 - 64bit Beta
With the Partitioning, Real Application Clusters, OLAP, Advanced Analytics
and Real Application Testing options
Build label: RDBMS_MAIN_LINUX.X64_151013
ORACLE_HOME: …/3775268204/oracle
System name: Linux
Node name: slc05kyr
Release: 2.6.39-400.211.1.el6uek.x86_64
Version: #1 SMP Fri Nov 15 13:39:16 PST 2013
Machine: x86_64
VM name: Xen Version: 3.4 (PVM)
Instance name: hm62
Redo thread mounted by this instance: 2
Oracle process number: 19
Unix process pid: 12656, image: oracle@slc05kyr (DIA0)
*** 2015-10-13T16:47:59.541509+17:00
*** SESSION ID:(96.41299) 2015-10-13T16:47:59.541519+17:00
*** CLIENT ID:() 2015-10-13T16:47:59.541529+17:00
*** SERVICE NAME:(SYS$BACKGROUND) 2015-10-13T16:47:59.541538+17:00
*** MODULE NAME:() 2015-10-13T16:47:59.541547+17:00
*** ACTION NAME:() 2015-10-13T16:47:59.541556+17:00
*** CLIENT DRIVER:() 2015-10-13T16:47:59.541565+17:00
2015-10-13T16:47:59.435039+17:00
Errors in file /oracle/log/diag/rdbms/hm6/hm6/trace/hm6_dia0_12433.trc (incident=7353):
ORA-32701: Possible hangs up to hang ID=1 detected
Incident details in: …/diag/rdbms/hm6/hm6/incident/incdir_7353/hm6_dia0_12433_i7353.trc
2015-10-13T16:47:59.506775+17:00
DIA0 requesting termination of session sid:40 with serial # 43179 (ospid:13031) on instance 2
due to a GLOBAL, HIGH confidence hang with ID=1.
Hang Resolution Reason: Automatic hang resolution was performed to free a
significant number of affected sessions.
DIA0: Examine the alert log on instance 2 for session termination status of hang with ID=1.
In the alert log on the instance local to the session (instance 2 in this case),
we see the following:
2015-10-13T16:47:59.538673+17:00
Errors in file …/diag/rdbms/hm6/hm62/trace/hm62_dia0_12656.trc (incident=5753):
ORA-32701: Possible hangs up to hang ID=1 detected
Incident details in: …/diag/rdbms/hm6/hm62/incident/incdir_5753/hm62_dia0_12656_i5753.trc
2015-10-13T16:48:04.222661+17:00
DIA0 terminating blocker (ospid: 13031 sid: 40 ser#: 43179) of hang with ID = 1
requested by master DIA0 process on instance 1
Hang Resolution Reason: Automatic hang resolution was performed to free a
significant number of affected sessions.
by terminating session sid:40 with serial # 43179 (ospid:13031)

Program Agenda
18/19c and beyond
Q&A
1
2
3
4
5
102

Oracle RAC 18c
• Manages hung database processes
– Detect & resolves
– Cross-layer hangs
• I.E: Hangs caused by a blocked ASM resource.
• Resolves deadlocks
• User defined control via PL/SQL
• Early Warning exposed via (V$ view)
103
Hang Manager
Database Member
Cluster
Uses ASM IO
Service
IO Service
ASM
Service
Session
Detect
Analyze
Evaluate
Hung?
Hang
Resolution

Diagnostic Service
Oracle Confidential – Highly Restricted 104
All data
aggregated in
one place
Real-time
overview of
infrastructure &
services
Fine-grained drill
down for
diagnosis

Initial Anomalous Events Ranked Anomalous Events
Oracle Confidential – Highly Restricted 105
Timestamp Correlation & Ranking
Full initial list of anomalous events
1. Sort the anomalous events in chronological order
2. keep tack of unique events and their first occurrence
3. Compare sequence of events to previous timeframes in the same collection
4. Prioritize unique events not seen previously in the collection

Oracle 19c
• Applied Machine Learning for
Database Diagnostics
– Efficient diagnosis using Machine
Learning
– Automatically performs
corrective actions to prevent
possible issues
– Provides simple alerts &
recommendations for issues that
require manual intervention
Oracle Domain Services Cluster
IO Service
ACFS
Services
ASM
Service
TFA
Service
Management
Service
RHP
Service
Shared ASM
Subject Matter
Expert
ASH
ML
Knowledge
Extraction
Model
Generation
Human
Supervision
Application
Optimized
Models
Feedback

Monitor
Detect
Review
Resolve
• Increase in number of resources in
the Global Resource Directory
(GRD)
• Resulting in higher wait times for
GRD
• Several solutions possible
– Is wait time due to high CPU load?
– Increase in number of LMS help?
– Increasing CR slaves help
– Increasing internal thresholds help?
Database Reliability Framework in Action

Database Reliability Framework Details
• Runs in LMHB Process
– Re-startable
– Non Fatal
• Relies on Metrics and Actions
Action Related metrics
report high cpu oracle instance
processes
cpu load, cpu threshold, bg
heartbeat, cpu load (global)
report high memory oracle instance
processes
memory load, memory threshold
kill instance memory hog memory load, memory threshold
enable rm plan cpu load, bg slowing (2)
switch process to elevated priority cpu load, bg slowing (3)
switch process from elevated priority cpu load, bg heartbeat
shrink or grow resource cache library cache pin waits
cap total processes at elevated priority cpu number
enable drm cpu load, drm checks
disable drm cpu load, drm checks
increase default number of lms cr blocks congested, current
blocks congested
Metric Scope
o/s memory Node Global
o/s load Node Global
bg heartbeat Global
DRM health check Global
library cache pin waits Local
CFIO waits Global
gc block lost Local
gc block busy Local

• Busy FG process(es) using CPU
• Potential upcoming memory
starvation
• LGWR constrained by CPU
• Too many RT processes
• Insufficient CR slaves
• DLM resource cache incorrectly
sized
• Control file IO (CFIO) stall
• v$ views
• v$gcr_metrics - details on all defined
metrics
• v$gcr_actions - details on all defined
actions
• v$gcr_log – metric/action history
summary log
• v$gcr_status – details on latest
metric/action status
109
Examples and DRF Views

• Increase the maximum number of
LMSs
– Based on System utilization (DRF)
• Each LMS will spawn a dedicated
CR slave
– Threshold of Rollback Changes
– Threaded CR slave in 18c
• Optimized for Multi core/thread architecture
• Remastering Slaves (RMV0..)
– Offloads heavy remastering work to
slaves
Cache Fusion Optimizations

Commit Cache
• Reduce Cache Fusion traffic for
remote undo header lookups
• Often becomes a bottleneck with
DML heavy OLTP/mixed workloads
• Remote undo header lookups are
needed for:
– Check if a transaction has committed
– Delayed block cleanout
111
0
400
800
1,200
1,600
2,000
Data
Blocks
Undo
Headers
Undo
Blocks
Others
# Block Transfers (thousands)
CR (Immediate) CR (Busy)
Current (Immediate) Current (Busy)

• Monitors for problems before
service disruption
– E.g HB for critical processes
• Detects the cause of problem
• Use collected data across all nodes
to identify root cause
– E.g. Waits on GRD
• Resolves the problem with minimal
disruption
– E.g Resize internal Structures
Introducing Database Reliability Framework
• Normal
• Busy
• Extremely Busy
Resource Utilization
• Type 1
• Type 2
• Type 3
Resource Types
Resources

• Undo Block RDMA-read
• In some workloads, more than half of remote reads are
for Undo Blocks to satisfy read consistency
– Undo Block RDMA-read uses RDMA to directly
and rapidly access UNDO blocks in remote
instances
• Commit Cache
– The Commit Cache maintains an in-memory
table on each instance which records the
commit time of transactions
– Remote LMS directly reads the commit cache
and sends back commit times for requested
transactions.
• Replaces having to send entire 8K transaction
table block
113
RAC Optimizations for Exadata
UNDOUNDO
RDMA RDMA

Fusion Block Transfer
1 2 3 4
Shadow Process LGWR
gc current block busy,
gc buffer busy acquire,
gc buffer busy release

• On Exadata, Oracle does not wait for the
log write notification
– Exadata ensures the log write completes before
changes to block on another instance commit,
guaranteeing durability
– Wait for Log I/O during transfer of hot blocks is
eliminated
– Up to 40% throughput and 33% response time
improvement in some heavily contended OLTP
workloads
• Storage software will ensure
correct ordering of writes
115
Smart Fusion Block Transfer
1. Issue log write
2. Wait for log
write completion
3. Transfer
block
Exadata Avoids I/O Wait confirmation
Storage

Continuous Feature Improvements
Lock Domain per PDB Utilize Bloom Filter to further
reduce Reconfiguration times
Utilize Database Reliability
Framework

Scalable Sequences
Continuous Application Availability
Oracle RAC Sharding
Cluster Domains
Cluster Health Advisor (CHA)
RAC Reader Nodes
Application Continuity (AC)
Oracle Flex ASM & Flex Clusters
Rapid Home Provisioning (RHP)
Oracle Quality of Service Management (QoS)
Policy-Based Cluster Management
Oracle RAC One Node & RACcheck
Oracle ASM Cluster File System (ACFS)
Oracle Grid Infrastructure (GI)
UCP and OCI Load Balancing Support for RAC
Cluster Verification Utility (CVU)
Cluster-Managed Services
Oracle Clusterware
Oracle Automatic Storage Management (ASM)
Oracle Real Application Clusters (RAC) Oracle 9i
Oracle RAC’s Journey into the Autonomous Database
20-years of continuous innovation*
Oracle 18c
* Documented features list is selective; 20 years include development time
117

Flex Cluster
Leaf nodes deprecated
Massive Parallel Query Oracle RAC
deprecated
Oracle RAC Reader Nodes
to be implemented on Hub nodes
Flex Cluster – Changes Down the Road
118

gridSetup and zip-based install
for Oracle Grid Infrastructure
NEW: RPM-based installs for the
Oracle Database and Oracle Client
ASM Management for
NFS-based Clusterware files
for easier management and thereby
better availability.
Separate Diskgroup for Grid
Infrastructure Management
Repository (GIMR)
allows for more flexibility during Grid
Infrastructure Installation
Better Management
$ORACLE_HOME/gridSetup.sh
Configure ASM on NFS
119

More Changes..
• Desupport of Direct File System Placement for Oracle Clusterware Files
– Introduced with Oracle Clusterware 12c Rel. 2 (12.2.0.1)
– Effective with Oracle Clusterware 18c
– Desupport revoked effective with Oracle Clusterware 19c
• Oracle Grid Infrastructure Management Repository (GIMR)
– Around since Oracle Grid Infrastructure 11g Release 2
– Automatic Installation of the GIMR introduced with Grid Infrastructure 12.1.0.2
– Separate diskgroup installation introduced with Grid Infrastructure 12c Release 2
– Automatic install revised for Oracle Grid Infrastructure 19c
• Plans foresee a GIMR installation outside of the Oracle Grid Infrastructure home for Standard Clusters
• Centralized GIMR hosting on a Domain Services Cluster (for Member Clusters) remains unchanged
120

Innovation on the Way to the Autonomous Database
• A proactive and automatic monitoring and correction framework
– Some functionality first introduced in Oracle RAC 12c
– Current framework used in Oracle RAC 18c; further enhancements planned for Oracle Database 19c
– Monitors various (currently ~50) metrics across different layers continuously
• Shares and considers information globally, but acts locally
– Detects problems before any disruption of service occurs
• v$ tables provide logs showing current system status and history of issues detected
– Identifies root cause accurately, based on current system situation
• Uses a combination of metrics to predict potential issues and identifies root cause(s)
– Resolves problems with minimum disruption, ideally before it happens
• Takes preventative action based on identified root cause
– Serializes actions across the cluster to minimize resolution impact
• Corrective actions are performed on per resource basis
Introducing Database Reliability Framework (DRF) (Working Name)
121

Patching Improvements
• OJVM is Oracle RAC rolling patch enabled with Oracle RAC 18c (18.4)
– Non-Java services are available at all times
– Java services are available all the time, except for a ~10 seconds brownout
• No errors are reported during the brownout
• Zero-Downtime Oracle Grid Infrastructure Patching (*18.3)
– Patch Oracle Grid Infrastructure without interrupting database operations
– Patches are applied out-of-place and in a rolling fashion with one node being patched
at a time while the database instance(s) on that node remain up and running
– Supported for Oracle RAC and RAC One Node clusters with two or more nodes
122

Undo RDMA-Read & Commit Cache
• Undo RDMA-Read (*18c)
– In some workloads, more than half of the remote
reads are for Undo Blocks to satisfy read consistency
– Undo Block RDMA-read uses RDMA to directly and
rapidly access UNDO blocks in remote instances
• Avoids having to activate a remote LMS to perform the operation
• Commit Cache (*18c)
– The Commit Cache maintains an in-memory, simple
hash table of recent transactions on each instance
– Remote LMS directly reads the commit cache and
sends back commit times for requested transactions
• Replaces having to send entire 8K transaction table block
• Customer test results:
– Up to 57% of the cache-fusion block traffic was eliminated
– cluster wait time down by 22%; DB CPU time down by 7%
UNDOUNDO
RDMA RDMA
Inst
1
Inst
3
Inst
2
123

The Road Ahead Leads into the Autonomous Database Cloud
• Future scalability & performance improvements
– Tailor to scaling well within Exadata dimensions (“scale linear across 64 nodes, not 200”)
– Are designed to meet ADB performance requirements and will grow as ADB enhances
– Will leverage RDMA technology for server-less communication
– Plan to use RoCE as the next-generation network for the cloud
• Details in MOS note “Oracle RAC Interconnect Protocols – Support and Roadmap (ID 2434852.1)”
– Will substitute storage access with network-based access to data on remote nodes
– Are likely to utilize NVM for storage on independent servers
• Future availability improvements
– Will focus on reducing re-configuration times (brownouts) further to come closer to “zero”
– Will provide even more ways to perform maintenance & admin tasks with no downtime
124

Troubleshooting Scenarios (1)
• Multicast Domain Name Service Daemon (mDNS(d))
– Used by Grid Plug and Play to locate profiles in the cluster, as well as by GNS to
perform name resolution. The mDNS process is a background process on Linux and
UNIX and on Windows.
– Uses multicast for cache updates on service advertisement arrival/departure.
– Advertises/serves on all found node interfaces.
– Log is GI_HOME/log/<node>/mdnsd/mdnsd.log
– Verify with mcasttest.pl , CVU , orachk
Cluster Startup Problem Triage

<?xml version="1.0" encoding="UTF-8"?>
<gpnp:GPnP-Profile Version="1.0" xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile" xmlns:gpnp="http://www.grid-pnp.org/2005/11/gpnp-profile"
xmlns:orcl="http://www.oracle.com/gpnp/2005/11/gpnp-profile" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.grid-
pnp.org/2005/11/gpnp-profile gpnp-profile.xsd" ProfileSequence="6" ClusterUId="b1eec1fcdd355f2bbf7910ce9cc4a228" ClusterName="staij-cluster" PALocation="">
<gpnp:Network-Profile><gpnp:HostNetwork id="gen" HostName="*">
<gpnp:Network id="net1" IP=”192.168.1.0" Adapter="eth0" Use="public"/>
<gpnp:Network id="net2" IP=”192.168.2.0" Adapter="eth1“ Use="cluster_interconnect"/>
</gpnp:HostNetworkcss"></gpnp:Network-Profile>
<orcl:CSS-Profile id=" DiscoveryString="+asm" LeaseDuration="400"/>
<orcl:ASM-Profile id="asm" DiscoveryString="" SPFile="+SYSTEM/staij-cluster/asmparameterfile/registry.253.693925293"/>
<ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#"><ds:SignedInfo><ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-
c14n#"/><ds:SignatureMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"> <InclusiveNamespaces xmlns="http://www.w3.org/2001/10/xml-exc-c14n#"
PrefixList="gpnp orcl xsi"/></ds:Transform></ds:Transforms> <ds:DigestMethod
Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/><ds:DigestValue>x1H9LWjyNyMn6BsOykHhMvxnP8U=</ds:DigestValue> </ds:Reference> </ds:SignedInfo>
<ds:SignatureValue>N+20jG4=</ds:SignatureValue> </ds:Signature>
</gpnp:GPnP-Profile>

• cssd agent and monitor
– Same functionality in both agent and monitor
– Run realtime with locked down memory, like CSSD
– Provides enhanced stability and diagnosability
– Logs are
• GI_HOME/log/<node>/agent/oracssdagent_root/oracssdagent_root.log
• GI_HOME/log/<node>/agent/oracssdmonitor_root/oracssdmonitor_root.log
• 12.1 – ORACLE_BASE/diag/node/agent/..

Missing Network Heartbeat (1)
• ocssd.log from node 1
• ===> sending network heartbeats other nodes. Normally, this message is output once every 5 messages (seconds)
• 2017-08-13 17:00:20.023: [ CSSD][4096109472] clssnmSendingThread: sending status msg to all nodes
• 2017-08-13 17:00:20.023: [ CSSD][4096109472] clssnmSendingThread: sent 5 status msgs to all nodes
• ===> The network heartbeat is not received from node 2 (rwsrac2) for 15 consecutive seconds.
• ===> This means that 15 network heartbeats are missing and is the first warning (50% threshold).
• 2017-08-13 17:00:22.818: [ CSSD][4106599328] clssnmPollingThread: node rwsrac2 (2) at 50% heartbeat fatal, removal in 14.520 seconds
• 2017-08-13 17:00:22.818: [ CSSD][4106599328] clssnmPollingThread: node rwsrac2 (2) is impending reconfig, flag 132108, misstime 15480
• ===> continuing to send the network heartbeats and log messages once every 5 messages
• ===> 75% threshold of missing network heartbeat is reached. This is second warning.

• ===> continuing to send the network heartbeats and log messages once every 5 messages
• ===> continuing to send the network heartbeats, but the message is logged after 4 messages
• ===> Last warning shows that 90% threshold of the missing network heartbeat is reached.
• ===> The eviction will occur in 2.49 seconds.
• 2017-08-13 17:00:34.841: [ CSSD][4106599328] clssnmPollingThread: node rwsrac2 (2) at 90% heartbeat fatal, removal in 2.490 seconds,
seedhbimpd 1
• ===> Eviction of node 2 (rwsrac2) started
• 2017-08-13 17:00:37.337: [ CSSD][4106599328] clssnmPollingThread: Removal started for node rwsrac2 (2), flags 0x2040c, state 3, wt4c 0
• ===> This shows that the node 2 is actively updating the voting disks
• 2017-08-13 17:00:37.340: [ CSSD][4085619616] clssnmCheckSplit: Node 2, rwsrac2, is alive, DHB (1281744040, 1396854) more than disk timeout of
27000 after the last NHB (1281744011, 1367154)

• ===> Evicting node 2 (rwsrac2)
• 2017-08-13 17:00:37.340: [ CSSD][4085619616](:CSSNM00007:) clssnmrEvict: Evicting node 2, rwsrac2, from the cluster in
incarnation 169934272, node birth incarnation 169934271, death incarnation 169934272, stateflags 0x24000
• ===> Reconfigured the cluster without node 2
• 2017-08-13 17:01:07.705: [ CSSD][4043389856] clssgmCMReconfig: reconfiguration successful, incarnation 169934272 with 1
nodes, local node number 1, master node number 1

• ocssd.log from node 2:
• ===> Logging the message to indicate 5 network heartbeats are sent to other nodes
• ===> First warning of reaching 50% threshold of missing network heartbeats
• 2017-08-13 17:00:26.213: [ CSSD][4073040800] clssnmPollingThread: node rwsrac1 (1) is impending reconfig, flag 394254, misstime 15460
• ===> Second warning of reaching 75% threshold of missing network heartbeats

• ===> Logging the message to indicate 4 network heartbeats are sent
• ===> Third warning of reaching 90% threshold of missing network heartbeats
• 2017-08-13 17:00:38.236: [ CSSD][4073040800] clssnmPollingThread: node rwsrac1 (1) at 90% heartbeat fatal, removal in 2.460 seconds, seedhbimpd 1
• ===> Eviction started for node 1 (rwsrac1)
• 2017-08-13 17:00:40.702: [ CSSD][4073040800] clssnmPollingThread: Removal started for node rwsrac1 (1), flags 0x6040e, state 3, wt4c 0
• ===> Node 1 is actively updating the voting disk, so this is a split brain condition
• 2017-08-13 17:00:40.706: [ CSSD][4052061088] clssnmCheckSplit: Node 1, rwsrac1, is alive, DHB (1281744036, 1243744) more than disk timeout of
27000 after the last NHB (1281744007, 1214144)
• 2017-08-13 17:00:40.706: [ CSSD][4052061088] clssnmCheckDskInfo: My cohort: 2
• 2017-08-13 17:00:40.707: [ CSSD][4052061088] clssnmCheckDskInfo: Surviving cohort: 1

• ===> Node 2 is aborting itself to resolve the split brain and ensure the cluster integrity
• 2017-08-13 17:00:40.707: [ CSSD][4052061088](:CSSNM00008:) clssnmCheckDskInfo: Aborting local node to avoid splitbrain.
Cohort of 1 nodes with leader 2, rwsrac2, is smaller than cohort of 1 nodes led by node 1, rwsrac1, based on map type 2
• 2017-08-13 17:00:40.707: [ CSSD][4052061088]###################################
• 2017-08-13 17:00:40.707: [ CSSD][4052061088] clssscExit: CSSD aborting from thread clssnmRcfgMgrThread
• 2017-08-13 17:00:40.707: [ CSSD][4052061088]###################################

Observations
1. Both nodes reported missing
heartbeats at the same time
2. Both nodes sent heartbeats to
other nodes all the time
3. Node 2 aborted itself to resolve
split brain
Conclusion
1. This is likely a network problem,
engage network team
2. Check OSWatcher output
(netstat and traceroute) and CPU
utilization on Node 2
• Configure private.net file, not
configured by default
3. Check CHMOS
4. Check system log

Voting Disk Access Problem (1)
ocssd.log:
===> The first error indicating that it could not read voting disk -- first message to indicate a problem accessing the voting disk
2017-08-13 18:31:19.787: [ SKGFD][4131736480]ERROR: -9 (
Error 27072, OS Error (Linux Error: 5: Input/output error
Additional information: 4
Additional information: -1)
)
2017-08-13 18:31:19.787: [ CSSD][4131736480](:CSSNM00060:) clssnmvReadBlocks: read failed at offset 529 of /dev/sdb8
2017-08-13 18:31:19.802: [ CSSD][4131736480] clssnmvDiskAvailabilityChange: voting file /dev/sdb8 now offline

====> The error message that shows a problem accessing the voting disk repeats once every 4 seconds
2017-08-13 18:31:23.782: [ CSSD][150477728] clssnmvDiskOpen: Opening /dev/sdb8
2017-08-13 18:31:23.782: [ SKGFD][150477728] Handle 0xf43fc6c8 from lib :UFS:: for disk :/dev/sdb8:
2017-08-13 18:31:23.782: [ CLSF][150477728] Opened hdl:0xf4365708 for dev:/dev/sdb8:
2017-08-13 18:31:23.787: [ SKGFD][150477728] ERROR: -9(
)

====> The last error that shows a problem accessing the voting disk.
====> Note that the last message is 200 seconds after the first message
====> because the long disktimeout is 200 seconds
2017-08-13 18:34:37.423: [ CSSD][150477728] clssnmvDiskOpen: Opening /dev/sdb8
2017-08-13 18:34:37.423: [ CLSF][150477728] Opened hdl:0xf4336530 for dev:/dev/sdb8:
2017-08-13 18:34:37.429: [ SKGFD][150477728] ERROR: -9(
)

====> This message shows that ocssd.bin tried accessing the voting disk for 200 seconds
2017-08-13 18:34:38.205: [ CSSD][4110736288](:CSSNM00058:) clssnmvDiskCheck: No I/O completions for 200880 ms for voting file
/dev/sdb8)
====> ocssd.bin aborts itself with an error message that the majority of voting disks are not available. In this case, there was only
one voting disk, but if three voting disks were available, as long as two voting disks are accessible, ocssd.bin will not abort.
2017-08-13 18:34:38.206: [ CSSD][4110736288](:CSSNM00018:) clssnmvDiskCheck: Aborting, 0 of 1 configured voting disks available,
need 1
2017-08-13 18:34:38.206: [ CSSD][4110736288]###################################
2017-08-13 18:34:38.206: [ CSSD][4110736288] clssscExit: CSSD aborting from thread clssnmvDiskPingMonitorThread
2017-08-13 18:34:38.206: [ CSSD][4110736288]###################################
Conclusion
The voting disk was not available, engage storage team

• Time synchronisation issue
• Cluster Time Synchronisation Services daemon
– Provides time management in a cluster for Oracle.
• Observer mode when Vendor time synchronisation s/w is found
–Logs time difference to the CRS alert log
• Active mode when no Vendor time sync s/w is found
Node Eviction Triage

• Agent return codes
– Check entry must return one of the following return codes:
• ONLINE
• UNPLANNED_OFFLINE
– Target=online, may be recovered failed over
• PLANNED_OFFLINE
• UNKNOWN
– Cannot determine, if previously online, partial then monitor
• PARTIAL
– Some of a resources services are available. Instance up but not open.
• FAILED
– Requires clean action
Agents stuck in undesirable states

• Important logs and traces
• 11.2 – Databases only use ADR
• Grid Infrastructure files in $GI_HOME/log/<node_name>/<component_name>
• $GI_HOME/log/myHost/cssd
• $GI_HOME/log/myHost/alertmyHost.log
• 12.1 – Grid Infrastructure and Database use ADR
• Different locations for Grid Infrastructure and Databases
• Grid Infrastructure
• Alert.log, cssd.log, csrd.log, etc
• Databases
• Alert.log, background process traces, foreground process traces
Automatic Diagnostic Repository (ADR)

NZOUG-GroundBreakers-2018 - Troubleshooting and Diagnosing 18c RAC

Empfohlen

Empfohlen

Weitere ähnliche Inhalte

Was ist angesagt?

Was ist angesagt? (20)

Ähnlich wie NZOUG-GroundBreakers-2018 - Troubleshooting and Diagnosing 18c RAC

Ähnlich wie NZOUG-GroundBreakers-2018 - Troubleshooting and Diagnosing 18c RAC (20)

Mehr von Sandesh Rao

Mehr von Sandesh Rao (20)

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

NZOUG-GroundBreakers-2018 - Troubleshooting and Diagnosing 18c RAC