Oracle Real Application Cluster 10g,11g Concept Included with ASM Concept

1. Oracle RAC & ASM
Osama Mustafa Page 1
RAC Concept
What is Real Application Cluster?
Cluster Consists of multiple interconnected servers that appears to user as one single database,
RAC allow instance to reside from one server to another in case of Failure. The below Figure
explain the basic setup for Real Application Cluster. Another definition you should be aware of
Node, which is unit contain CPU and Memory called Server.
Before Start talking about Oracle RAC, what is the advantage of using Oracle RAC?
 Scalability: multiple nodes allow cluster database to scale by single node database.
 Availability: if any nodes failure other nodes in cluster, clients can continue working
without any effects.
 Manageability: more than one database can be handled by oracle Cluster ware.

2. Oracle RAC & ASM
Osama Mustafa Page 2
There are two kinds of cluster active/active and active/passive
Acitve/passive
In this setup usually we have two nodes, one of the nodes are available (active) and the other one
is not (passive), Oracle software should be on Shared storage , and only run on one node ( active
) in case of failure the cluster convert the shared storage to passive node in that case active node
now is passive and the passive node is now active.
There is number of Third Party software Corporation support this kind of cluster such as
Microsoft, Linux. Usually it’s called OS cluster.
Active/Active
In this kind of setup Oracle instance run concurrently on both server and client access to both
server at the same time, the instance should communicate with other node to ensure (heartbeat)
both server are available but in case any of server goes down the other server can handle the
workload, the benefits of active/active workload can be shared between servers.
Some other concept I would like to mention here, Shared Disk and Non Shared disk in oracle
Real Application Cluster Installation, there’s two option when you install Oracle RAC Nothing
to Share means database is configured as number of nodes but each node has it own private disk
only thing shared will be is Database, option number two is shared everything all nodes access to
same data in this case if any of node failure no missing or loss of data.
When I am talking about Real Application Cluster I already mean High Availability (HA) which
is the same purpose of Oracle RAC ensure system or components is continuously operational for
long time. RAC can be complex and not easy to manage, design, install, configure and tune and
finally not to forget the license which is cost lot. The reason behind deploy RAC is different
from company to another some users implement RAC to ensure HA, Other user implement RAC
to manage workload. You can use alternative solution of RAC if you don’t have that much of
money such Data Guard or third party Software which I already mention it.
Difference RAC & NON-RAC
I would like to mention that share the same concept, background Process in RAC found in Single
Node (PMON,SMON,DBWR,ARCn,CKPT) , in RAC system Each Instance has its own System
Global Area and Redo log , This is if we are talking about Structure but Oracle Cluster has some
Daemons Event Manager Daemon ( emvd ) ,Oracle Cluster synchronization Service Daemon
(cssd ) and finally cluster ready services daemon ( crsd ).notice both single instance and rac
instance can be managed by Enterprise Manager.

3. Oracle RAC & ASM
Osama Mustafa Page 3
Another Component should be mention when talking about RAC Oracle Cluster Registry (OCR)
shared storage should be available for OCR the benefits of using OCR is Store cluster
configuration such as names, services and instance, Voting Disk also should be on shared storage
is required to determine the nodes that are currently available you can search for process
(CSSD).
Check OCR using crscheck or using crsctl query css votedisk
Node Application In Real Application Cluster
Is services automatically configured when On installing Phase, and enable by default when node
is booted.
Listener
Listener is process runs on each node and the purpose for this process is listening to incoming
connection from clients and redirected to RAC instance, since there’s two type of connection
dedicated and shared listener is responsible for managing workload.
Oracle Notification Services ( ONS )
Started by the CRS cluster ware as part of the nodeapps. There is one ons daemon started per
clustered node. Receive a subset of published cluster ware events via the local evmd and
racgimon cluster ware daemons and forward those events to application subscribers and to the
local listeners.
There’s Two Type Of this kind of node app process Fast Application Notification ( FAN ) Which
is Used to notify application about any changes in configuration for example start/stop Services.
On other hand we Fast Connection Failover (FCF) The main purpose for this process is to
prevent any new connection for Failed nodes or instance.
Virtual IP ( VIP )
In Oracle 10gR1 Clients Can access to database Rac using VIP no More Public, VIP is Ip
address assign for each node belong to same Subnet mask For public, In case any Faiover VIP
Connect to Another Node.
Global Services Daemon ( GSD )
This process allows clients including SRVCTL and DBCA to execute administrative Command.

4. Oracle RAC & ASM
Osama Mustafa Page 4
How to Check Oracle RAC Process is UP :
# ps –ef | grep evmd
#ps –ef | grep cssd
#ps –ef | grep crsd
I can check ASM ( I will talk later about it )
#ps –ef | grep asm
Also you Can Check http://support.oracle.com notes :
Script to Collect RAC Diagnostic Information (racdiag.sql) [ID 135714.1]
Which give you script to collect health Status for RAC
Or Another way
#Crsctl Check crs
Or using SRVCTL Command
Check Database Status
Srvctl status database –d <Database-name>
Check status for Instance
Srvctl status instance –d <Database-name> -I <Instance-1> ,<instance-2>
you can check nodeapps
Srvctl status nodeapps –n <node-name>
Check All Resource
Crs_stat –t

5. Oracle RAC & ASM
Osama Mustafa Page 5
Trouble Shoot Oracle RAC:
 OCR Failure
 Vote Failure
 Reboot Issue
 Network Issue
 Rac Resource Offline
To solve any Issue in Oracle RAC you need to Look at the logs first which is Located in
$CRS_HOME/log/nodename/
Also Some Useful http://support.oracle.com notes for troubleshoot:
Diagnosing CRS-215 Error When Starting Clusterware Resources [ID 781632.1]
Troubleshooting when srvctl can't start RAC instance, but sqlplus can start it [ID 844272.1]
Another issue you can face with Oracle RAC which is Resource offline, as mentioned before you
can check it by crs_stat –t and usually the error “CRS-0215 could not start resource %”
You can try start this resourcr using srvctl command
Srvctl start nodeapps –n <resourcre-name>
Is this not working you should go further more than this logs will tell what exactly you have.
Failed/Corrupted OCR and Voting Disk
RAC: Frequently Asked Questions [ID 220970.1]
How to Replace a Corrupt OCR Mirror File [ID 317628.1]
OCR usually backup automatically in oracle rac and can restore so the issue would be in voting
disk in Linux you can use dd command to backup voting disk, and for windows use copy
command.
And notice that if you don’t have backup you need to reinstall CRS and this will cost you time
and Loss of money in same cases.
To restore OCR on RAC we use ocrconfig utility

6. Oracle RAC & ASM
Osama Mustafa Page 6
OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE) [ID 428681.1]
The backup path for OCR
$CRS_HOME/cdata/crs
#ocrconfig –showbackup
#ocrconfig –restore <backup-file-name>
Reboot issue in rac happened for different Cause maybe network,hardware,storage or even rac
process http://support.oracle.com note will help you to understand reboot
Troubleshooting 10g and 11.1 Clusterware Reboots [ID 265769.1]
Because of this you usually need to check matrix certification tab on support.oracle.com check
your network before starting setup process and health check for storage .
Monitor Oracle Real Application Cluster
 Automatic Workload Repository (AWR)
o Collect Database statistics using Snapshot
o Check Database Performance And SQL time
o You can generate AWR using Command Line
 @$ORACLE_HOME/rdbms/admin/awrrpt.sql
 @$ORACLE_HOME/rdbms/admin/awrrpti.sql
 Automatic Database Diagnostic Monitor (ADDM)
o Analyses data that captured by AWR
o Generate ADDM using Command Line
 $ORACLE_HOME/rdbms/admin/addmrpt.sql
 Enterprise manager
o Manage RAC database
o Display Performance as Graphical Interface
o Display Result for AWR,ADDM and ASH
 Third Party Tools
o OS Command Such as top
o Spot Software
 Active Session history (ASH)
o Help to solve simple performance issue.
o All active session will be generate every Second.

7. Oracle RAC & ASM
Osama Mustafa Page 7
As conclusion you need to use EM to monitor oracle rac database and diagnose performance
Issue this will lead to identify the root of cause issue and try to solve it using one of the above
tools provided by oracle.
Automatic Storage management
First time ASM technology introduced in Oracle 9.0.1 By Oracle OMF (Oracle Managed Files)
Features which automated the creation for database files such as data files, Redo log, and control
file, you can manage table space without reference to filename, ASM Supported Single and RAC
databases.
ASM Cam manage many type of database files for example : data files , temp files , control files,
online redo logs, archive logs and RMAN backup sets, later in newer version it can supported
parameter files , flashback logs , and change tracking files.
It’s designed to allow DBA (Database Administrator) to perform most disk management tasks; in
this case ASM has more advantage than LVM and RAID, Another Benefits for ASM For More
complexity Site which is managing large number of disk and you can do that by assign disk to
specific group, including to stripping and mirroring ASM Can Perform some other task such as
increasing reliability and performance By Default ASM Implements two-way mirroring to
ensure redundant copy of data therefore for critical data you can implement three way-mirroring,
another benefits for ASM which is provide mirroring at file level which give DBA more control.
Increasing Read and Write performance in ASM by strip mechanism by data across multiple
disk, and the stripes size control by database files, and as I mentioned earlier ASM Support
Single Instance and RAC database.
In Oracle Database 10gR1 ASM supported only Database 10gR1 check this example, if we have
database version 10.1.0.1 and ASM version 10.1.0.3 then ASM instance will revert to earlier
functionality , on the other hand if ASM version 10.1.0.2 and Database 10.1.0.3 then ASM will
continue using 10.1.0.2 functionality.
In Oracle 10gR2 this is totally different, ASM support different Database version, Oracle 10.2
ASM will support Oracle database 10.1 without losing any Functionality.

8. Oracle RAC & ASM
Osama Mustafa Page 8
How to Install ASM
Before you start install any Oracle Products you have to look for Recommendation, In ASM
Oracle Corporation recommended to install ASM binary in separate Oracle Home Why? If we
are not going to use this approach patch & upgrade will be hard. Otherwise Separate Home will
be more flexibility to perform patching and Upgrading.
Another recommendation for Installing ASM Binary from Oracle which is when you Install
ASM for production environment use shared storage for the following reasons:
 Upgrade and Patching Become Easier Only Once.
 One Copy of binary Needs to be copy
 Disk space for binary reduced.
Check https://oss.oracle.com to install the required package for ASM depend on your Platforms :
 Oracleasm
 Oracleasmlib
 Oracleasm-support
To install them use the below command :
Rpm –ivh <Package-Name>
When you finished installing the above package new device created for ASM called
/dev/oracleasm
And you can manage ASM using the below Command
[root@test root] # /etc/init.d/oracleasm <option>
Or
[root@test root] # service oracleasm <option>
The Options that can be used with the above command:
 Start : Start ASM services
 Stop : Stop ASM services
 Status : Print the ASM Services Status
 Configure : configure ASM
 Createdisk: Create ASM disk

9. Oracle RAC & ASM
Osama Mustafa Page 9
 Scandisks : search for ASM disks
 Querydisk: Query the current status of ASM disk
 Deletedisk : delete an ASM disk
 Restart : Restart ASM Services
 Enable : Enable ASM services
 Disable : Disable ASM services
All the above option will be described:
Start
[root@test root] # /etc/init.d/oracleasm start
stop
[root@test root] # /etc/init.d/oracleasm stop
Status
[root@test root] # /etc/init.d/oracleasm status
Checking if ASM is loaded: [OK]
Checking if /dev/oracleasm is mounted: [OK]
Configure
[root@test root] # /etc/init.d/oracleasm configure
# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: oracle
Default group to own the driver interface []: oinstall
Start Oracle ASM library driver on boot (y/n) [n]: y

10. Oracle RAC & ASM
Osama Mustafa Page 10
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
Createdisk
Note On RAC (Real Application Cluster) these disk should be on Shared Storage using raw
device or logical partition, when you are using raw device make sure it’s identical on each node,
and regarding to htts://support.oracle.com note number 578455.1 raw device is un-supported in
11gR2, so another option are available now such as ASM,OCFS and OCFS2, also notice that
you need to create ASM disks only on one node, on the second one you should only able to run
scandisks attribute.
# /etc/init.d/oracleasm createdisk disk1 /dev/sdb1
Marking disk "disk1" as an ASM disk: [ OK ]
# /etc/init.d/oracleasm createdisk disk2 /dev/sdc1
Marking disk "disk2" as an ASM disk: [ OK ]
# /etc/init.d/oracleasm createdisk disk3 /dev/sdd1
Marking disk "disk3" as an ASM disk: [ OK ]
# /etc/init.d/oracleasm createdisk disk4 /dev/sde1
Marking disk "disk4" as an ASM disk: [ OK ]
Scandisks
After creating Disk you should run this command on the second node/nodes
# /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks: [ OK ]

11. Oracle RAC & ASM
Osama Mustafa Page 11
Listdisks
You need to verify the disk, this command show you the list of disks you have on ASM
# /etc/init.d/oracleasm listdisks
Disk1
Disk2
Disk3
Disk4
Querydisk
You can query the status disks using querydisk as the example :
# /etc/init.d/oracleasm querydisk disk1
Disk “disk1” is a valid ASM disk
Deletedisk
You can delete existing disk using
# /etc/init.d/oracleasm deletedisk disk1
Removing ASM disk “Disk1”
ASM Administration Using ASMCMD
Asmcmd is new features in Oracle 10gr2 and above, ASMCMD utility allows you to inspect the
files within file system with very limited administration and modifications, and to use you should
export/set ORACLE_SID first to ASM instance as the below :
Export ORACLE_SID=+ASM1
By doing that new connection established, and the user should be member of SYSDBA group,
the below command supporting for ASMCMD
 Cd
 Du
 Find

12. Oracle RAC & ASM
Osama Mustafa Page 12
 Help
 Ls
 Lsct
 Lsdg
 Mkalias
 Mkdir
 Pwd
 Rm
 Rmalias
The below describe some of the above command:
Cd
This commands same as Linux, allow you change current directory but this time within
ASMCMD
Cd <dir>
For example
Cd +DISKGROUP1
Du
Display the total amount of space
Du [dir]
For example
Du +DISKGROUP1
Used_MB Mirror_used_MB
500MB 700MB
Find
Allow you search inside file system,you can start with directoy which may include wildcards and
file name pattern
Find [-t type] <dir> <pattern>
For example :
Find +DISKGROUP1/orcl/redo*

13. Oracle RAC & ASM
Osama Mustafa Page 13
+DISKGROUP1/orcl/redolog01.log
+DISKGROUP1/orcl/redolog02.log
+DISKGROUP1/orcl/redolog03.log
Help
Display summary of the commands available only in ASMCMD
Ls
List come or all files and their attributes, if you change the attribute output will be different each
time
Lsct
Lists information about current Oracle ASM clients.
lsct data
DB_Name Status Software_Version Compatible_version Instance_Name Disk_Group
+ASM CONNECTED 11.2.0.2.0 11.2.0.0.0 +ASM DATA
ASMVOL CONNECTED 11.2.0.2.0 11.2.0.0.0 +ASM DATA
Monitoring ASM
There’s two type of views to monitor ASM dynamic Performance view ( v$) and fixed view
(X$), notice both of these view are available for ASM and database instance.
Dynamic Performance Views
V$ASM_File
All ASM files belong to disk group mounted by ASM listed in this view.
V$ASM_DISK
List all disks discovered by ASM instance.
V$ASM_DISKGROUP
View lists all disk group discovered by ASM.

14. Oracle RAC & ASM
Osama Mustafa Page 14
V$ASM_CLIENT
List all client database access to disk group.
V$ASM_ALIAS
List all aliases in all currently mounted disk group.
V$ASM_DISK_STAT
Same as V$ASM_DISK but if new disk added it will not discover it.
Fixed Views
View Name Usage
X$KFMTA ASM template
X$KFKID ASM Disk Performance
X$KFGRP ASM disk Group
X$KDSK ASM Disks
X$KGMG ASM Operations
Check ASM
#ps –ef | grep asm
Using SQL PLUS
Sqlplus / as sysdba
Select name,path,state from v$ASM_DISK;