5. OEM High Availability Levels
Level Description Load Balancer Required Cost
1 OMS and Repository on separate hosts.
No redundancy.
No $
2 OMS installed on shared storage with VIP used in
active/passive failover.
Repository Database using local Data Guard.
No $$
3 Multiple OMSs in active/active configuration.
Repository using RAC Database with local Data Guard
Yes, At primary site $$$
4 Primary OMS in active/active configuration with RAC
Repository database. Standby OMS at DR site in
active/active configuration. Standby RAC database at
DR site
Yes. At pimary and standby
sites
$$$$
6. Level 2 – Active/Passive HA
• A minimum of 2 Servers Required
• OMS binaries installed on shared filesystem
• NFS/OCFS2/DBFS/ACFS
• OMS can run on one node in cluster at any given
time
• Data Guard for Management Repository
9. Oracle Clusterware Setup
• Clusterware can be used to create/manage VIP
• 11.2+ uses appvipcfg for VIP
<GRID_HOME>/bin/appvipcfg create -network=1
-ip=192.168.1.0
-vipname=omsvip
-user=root
• VIP can be created on non-default network
• In Oracle 12c Flex Clusters app vips can be create on
leaf nodes
• Allow Oracle Grid Infrastructure software owner (e.g.
grid) to run the script to start the VIP.
<GRID_HOME>/bin/crsctl setperm resource omsvip -u user:grid:r-x
10. Oracle Clusterware Setup
• Start the VIP as the GI owner e.g. grid
<GRID_HOME>/bin/crsctl start resource omsvip
• Check the status of the VIP
<GRID_HOME>/bin/crsctl status resource omsvip
The status of the output should be similar to the following:
NAME=omsvip
TYPE=app.appvip_net1.type
TARGET=ONLINE
STATE=ONLINE on oms1
11. Oracle Clusterware Setup
• Check if virtual hostname and VIP are resolvable
nslookup <omsvip>
• Also do a reverse lookup of the IP address.
nslookup <virtual IP address>
• Verify that the IP address returned from the nslookup
output is running on the OMS host.
ifconfig –a | grep <virtual IP address>
12. OEM Installation
• Create ORACLE_HOME for the OMS on the shared
storage on all nodes in the cluster
mkdir –p /u01/app/oms_share
• Create Oracle Inventory directory under ORACLE_HOME
for the OMS on all nodes
mkdir /u01/app/oms_share/oraInventory
• Create the inventory pointer in the oraInventory directory
vi oraInst.loc
inventory loc=/u01/app/oracle/oms_share/oraInventory
inst_group=oinstall
14. OEM Failover
• To manually relocate the VIP to another host in the
cluster issue the following command.
$crsctl relocate res omsvip
CRS-2673: Attempting to stop 'omsvip' on 'oms1'
CRS-2677: Stop of 'omsvip' on 'oms1' succeeded
CRS-2672: Attempting to start 'omsvip' on 'oms2'
CRS-2676: Start of 'omsvip' on 'oms2' succeeded
• Check if the IP address associated with the VIP is
running on the relocated host.
ifconfig –a|grep <vip>
15. OEM Failover
• Establish IP on failover server (done through Clusterware)
• Start listener (if part of same failover group)
• Start database (if required)
• Set the ORACLE_HOSTNAME environment variable to the
virtual hostname. Continuing with our example we use the
command below.
export ORACLE_HOSTNAME=omsvip.example.com
• Start the OMS on the new node
$OMS_HOME/bin/emctl start oms
16. Add OEM Clusterware resource
• OEM can be added as Clusterware resource
• Administrator managed (static, 2 – nodes)
• Policy managed (dynamic, > 2 nodes)
• Handled through Agent Framework
• C/C++
• Create Action Script
• Store on shared storage
• Specify START/STOP/CHECK/ABORT routines
17. Example
$ crsctl add server pool oem_sp -attr
"PARENT_POOLS=Generic, SERVER_NAMES=oms1 oms2”
$ crsctl add resource oem -type cluster_resource -attr
"ACTION_SCRIPT=/opt/cluster/scripts/oem12c.scr,
PLACEMENT='restricted',
SERVER_POOLS=oem_sp,
CHECK_INTERVAL='30',
RESTART_ATTEMPTS='2',
START_DEPENDENCIES='hard(omsvip)',
STOP_DEPENDENCIES='hard(omssvip)'"
18. Clusterware resource
• Oracle 12c introduced a new generic_application resource
type
crsctl add resource oem -type generic_application -attr
"START_PROGRAM='$OMS_HOME/bin/emctl start oms',
STOP_PROGRAM='$OMS_HOME/bin/emctl stop oms',
CLEAN_PROGRAM='$OMS_HOME/bin/emctl stop oms –all
-force',
CHECK_PROGRAM='$OMS_HOME/bin/emctl status oms'
19. Management Repository
• Can be configured using Data Guard with Fast-Failover
• Also can be configured with RAC/RAC One Node
• Storage becomes SPOF
• Listeners should be reachable from all nodes in cluster
• Database should be started before starting OEM
20. Configure OEM with Data Guard Repo
Create database service for fast failover.
23. References
• How to Configure Grid Control OMS in Active/Passive
CFC Environments failover / HA (Doc ID 405642.1)
• How to Configure the OMS Connect String when
Repository is in a Dataguard setup [ID 1328768.1]
• Oracle® Clusterware Administration and Deployment
Guide12c Release 1
24. COLLABORATE14.IOUG.ORGCOLLABORATE14.IOUG.ORG
• Network with 5,000+ database and
application professionals
• 5 days of in-depth education built by
users, for users
• Complimentary Pre-Conference
Workshops for IOUG registrants
April 7 – 11, 2014
The Venetian Resort,
Las Vegas, NV
Attend for free!
IOUG’s Call for Speakers is now open
collaborate14.ioug.org/call-for-speakers
25. Communities
Training Close to HomeTraining Close to Home
Maximum Availability Architecture
with Oracle’s Larry Carpenter and Joe Meeks
October 15: Milwaukee, WI
October 16: Rochester, NY
Performance and Internals with Craig Shallahamer
November 6: Chicago, IL
November 8: Atlanta, GA
Get more information at www.ioug.org/masterclass
or visit the IOUG kiosk in Moscone West, 2nd
floor
IOUG Master Class Series Returns
Coming soon to a city near you!
One Day Interactive Training Events to meet your
educational needs
26. Come See IOUG at the User Group PavilionCome See IOUG at the User Group Pavilion
Win a Free Registration to COLLABORATE 14!
Stop by the IOUG kiosk in the User Group Pavilion in
Moscone West, 2nd floor, to pick up a lucky poker chip and
see if you are a winner of a free registration to
COLLABORATE 14 in Las Vegas!
Each component within the Enterprise Manager architecture should be made highly available to enable a complete High Availability configuration. The main components to be considered are (see Figure 13-1): Enterprise Manager Agent - Communicates with and sends metrics to the OMS Management Server - the heart of the Enterprise Manager Repository -Stores persistent data from the monitored targets. Software library - Stores files for patching, provisioning and agent and plug-in deployment
Different levels of high availability can be configured for each component with varying levels of complexity and cost. When considering your high availability requirements, there should be minimal trade-offs in cost, complexity, performance and data loss. Generally, the complexity and level of high availability are proportional to each other.
Also called a cold-failover cluster (CFC) In order to reduce OMS downtime during planned or unplanned outage, some redundancy should be introduced into the configuration. A level 2 configuration uses a shared filesystem for the management service to achieve an active/passive or cold failover cluster solution. The filesystem is shared between two or more hosts and is only active on one host at a time. The shared filesystem for the OMS can be installed on a general-purpose cluster file system including NFS, Oracle Cluster File System (OCFS2), and Oracle Automatic Storage Management (ASM) Cluster File System (ACFS). If NFS is used as the shared storage, then ensure the correct mount options are set in /etc/fstab (/etc/filesystems on AIX) to prevent potential I/O issues. Specifically, the rsize and wsize should be set. The example below shows and entry in the /etc/fstab file on a Linux server where the NFS share is mounted on a filer named filer1 under the /vol1/oms_share directory. filer:/vol1/oms_share /u01/app/oms_share nfs rw,bg,rsize=32768,wsize=32768,hard,nointr,tcp,noac,vers=3,timeo=600 0 0 Binaries for the OMS along with the inventory should be installed on the shared filesystem. Setup the virtual hostname and IP address (VIP) using Oracle Clusterware or third-party software and hardware. Failover is achieved by using the virtual hostname for the OMS along with a unique IP address which resolves to the hostname.
Also called a cold-failover cluster (CFC) In order to reduce OMS downtime during planned or unplanned outage, some redundancy should be introduced into the configuration. A level 2 configuration uses a shared filesystem for the management service to achieve an active/passive or cold failover cluster solution. The filesystem is shared between two or more hosts and is only active on one host at a time. The shared filesystem for the OMS can be installed on a general-purpose cluster file system including NFS, Oracle Cluster File System (OCFS2), and Oracle Automatic Storage Management (ASM) Cluster File System (ACFS). If NFS is used as the shared storage, then ensure the correct mount options are set in /etc/fstab (/etc/filesystems on AIX) to prevent potential I/O issues. Specifically, the rsize and wsize should be set. The example below shows and entry in the /etc/fstab file on a Linux server where the NFS share is mounted on a filer named filer1 under the /vol1/oms_share directory. filer:/vol1/oms_share /u01/app/oms_share nfs rw,bg,rsize=32768,wsize=32768,hard,nointr,tcp,noac,vers=3,timeo=600 0 0 Binaries for the OMS along with the inventory should be installed on the shared filesystem. Setup the virtual hostname and IP address (VIP) using Oracle Clusterware or third-party software and hardware. Failover is achieved by using the virtual hostname for the OMS along with a unique IP address which resolves to the hostname.
Also called a cold-failover cluster (CFC) In order to reduce OMS downtime during planned or unplanned outage, some redundancy should be introduced into the configuration. A level 2 configuration uses a shared filesystem for the management service to achieve an active/passive or cold failover cluster solution. The filesystem is shared between two or more hosts and is only active on one host at a time. The shared filesystem for the OMS can be installed on a general-purpose cluster file system including NFS, Oracle Cluster File System (OCFS2), and Oracle Automatic Storage Management (ASM) Cluster File System (ACFS). If NFS is used as the shared storage, then ensure the correct mount options are set in /etc/fstab (/etc/filesystems on AIX) to prevent potential I/O issues. Specifically, the rsize and wsize should be set. The example below shows and entry in the /etc/fstab file on a Linux server where the NFS share is mounted on a filer named filer1 under the /vol1/oms_share directory. filer:/vol1/oms_share /u01/app/oms_share nfs rw,bg,rsize=32768,wsize=32768,hard,nointr,tcp,noac,vers=3,timeo=600 0 0 Binaries for the OMS along with the inventory should be installed on the shared filesystem. Failover is achieved by using the virtual hostname for the OMS along with a unique IP address which resolves to the hostname.
It’s recommended to use the appvipcfg utility in Oracle Clusterware 11gR2 to create application VIPs. The VIP is created with a set of pre-defined settings suitable for an application VIP such placement policy and the failback option. The default value of the failback is set to 0, which means that the VIP and its dependent resources will not automatically fail back to the original node once it becomes available again. VIP can be created on the non-default network (default ora.net1.network) Non-default network should be created with srvctl add network command
A VIP can be created in the same was any other Clusterware resource. However, it is recommended to use the appvipcfg utility in Oracle Clusterware 11gR2 to create application VIPs. The VIP is created with a set of pre-defined settings suitable for an application VIP such placement policy and the failback option. The default value of the failback is set to 0, which means that the VIP and its dependent resources will not automatically fail back to the original node once it becomes available again.
The virtual hostname is defined in DNS, and should resolve to the application VIP address created using the steps above. Check if the virtual hostname and VIP are resolvable using nslookup or the dig command. $ nslookup omsvip This should resolve to a unique IP address of the virtual hostname on every node in the cluster. Also do a reverse lookup of the IP address. $nslookup <virtual IP address> Verify that the IP address returned from the nslookup output is running on the OMS host. ifconfig –a|grep <virtual IP address>
Install the OMS on the first host by following the installation steps as described in the Oracle Enterprise Manager Cloud Control 12c Basic Installation Guide. You only need to complete the installation once. Since the location is shared, the binaries will be accessible from another host that shares the filesystem.
The ORACLE_HOSTNAME should be also set when starting the OEM on each node in the cluster. This should be the same as the Virtual Hostname defined in DNS for the VIP.
Once the OMS has been successfully installed and is up and running, if the host were to go down then the VIP will be automatically relocated to another node. The management service can then be manually started on any remaining node in the cluster on which the VIP is running.
Oracle Clusterware can be configured to fully manage the OMS by creating start, check, stop, clean and abort routines that tell it how to operate on the OMS.
You must decide whether to use administrator or policy management for the application. Use administrator management for smaller, two-node configurations, where your cluster configuration is not likely to change. Use policy management for more dynamic configurations when your cluster consists of more than two nodes. For example, if a resource only runs on node 1 and node 2 because only those nodes have the necessary files, then administrator management is probably more appropriate. An action script is a shell script (a batch script in Windows) that a generic script agent provided by Oracle Clusterware calls. An application-specific agent is usually a C or C++ program that calls Oracle Clusterware-provided APIs directly.
To add the OEM server as a resource that uses a named server deployment, assume that you add the resource to a server pool that is, by definition, a sub-pool of the Generic server pool. You create server pools that are sub-pools of Generic using the crsctl add serverpool command. These server pools define the Generic server pool as their parent in the server pool attribute PARENT_POOLS. In addition, they include a list of server names in the SERVER_NAMES parameter to specify the servers that should be assigned to the respective pool.
generic_application cluster type can model any application requiring high availability without specifying action scripts
If you like what you heard today, you should come see me speak at COLLABORATE 14 – IOUG Forum 1,000 customer sessions, very similar to what you see on User Group Sunday at OpenWorld Includes complimentary hands on labs and pre-conference workshops for IOUG attendees COLLABORATE 14 is April 7-11 in Las Vegas!
IOUG Master Class brings the best content from COLLABORATE 13 to a city near you Use these one-day training events to train your entire team at an affordable price! Group discounts available, visit the IOUG kiosk for more information
Visit IOUG in the User Group Pavilion, on the 2 nd floor of Moscone West to pick up your poker chip and win a prize Don’t forget to drop off your speaker card from this session to help me win a prize!