2. Mike Gangler
• Ann Arbor, Michigan
• Database Specialist at Ithaka.org – aka “JSTOR”
• Multi-platform and Database DBA
• Specializing in Performance, Tuning and management of databases and
environments
• Working With Oracle since Versus 4 ….
• Board of Directors for SEMOP (Southeast Michigan Oracle Professionals),
MOUS (Michigan Oracle User Summit, and charter member (1993) of the
Board of Directors of the International Oracle Users Group(1OUG).
Slide 2
3. What this presentation will cover
• EM12 Architecture High Availability options
• Setting up an Level 3 MAA Environment
• Installation of a Standby OMS – Silent Mode
• If Time---Notifications Best Practices
Slide 3
4. Audience Experience
• Grid Control Experience
• Used the old 9i / Workgroup version ?
• 10G Grid Control
• 11G Grid Control
• 12C Cloud Control
Slide 4
6. Presentation Goals
• Goals
• Learn the Different MAA architectures
• How to run a installation of a Standby OMS in
silent Mode
• Notifications Tips / Best Practices
Slide 6
7. Ithaka’s Mission
aka JSTOR
ITHAKA is a not-for-profit organization dedicated to
helping the academic community take full advantage
of rapidly advancing information and networking
technologies. We serve scholars, researchers, and
students by providing the content, tools, and services
needed to preserve the scholarly record and to
advance research and teaching in sustainable ways.
We are committed to working in collaboration with
other organizations to maximize benefits to our
stakeholders.
www.ithaka.org - www.jstor.org
Slide 7
8. EM12C Architecture
Options
• Level 1 – Single Site – No Failover
• Level 2 – Single Site
• 2 OMS – Active / Passive Mode
• VIP Failover
Slide 8
9. EM12C Architecture
Options
• Level 3
• Single Site
• Multiple OMS (Active/Active)
• Load Balancer
• RAC Data Guard Primary / Standby DB
Slide 9
11. EM12C Architecture
Suggestions
• Setup an additional / separate OMS for Dev/QA targets
• Setup Level 3.5 Maximum Availability Architecture (MAA)
• Multiple Sites
• Multiple OMS (Primary / Standby)
• Oracle Data Guard with Fast Start Failover setup.
• Using VIP’s / DNS alias help with data Guard rollover.
• Use Load Balancer (optional) for OMS.
Slide 11
13. Monitoring at JSTOR
• Production Oracle Databases – 25 – 24x7
• Development Oracle Databases – 27 – 14x5
• Microsoft SqlServer Databases - 21 – 14x5
• MySql Databases – 9 – 14x5
• Total 240 Production Targets (Agents, hosts, db, etc.)
• Total 276 Dev/QA Targets (Agents, hosts, db, etc.)
Slide 13
14. Ithaka OMS Layout
DATACENTER DATA DATACENTER
1 CENTER 2 3
DNS
DataCenter 1 DataCenter 2
Secondary Primary OMS
OMS
DNS
Datacenter1
Standby Primary
DataGuard DB
DB
Slide 14
15. Pre-requisites for High Availability
• Setup VIP or DNS Alias for multiple
sites.
• Create Data Guard database on other
site.
• NFS Shared Library
• Primary OMS Setup
• Plugins Installed on Primary Site
Slide 15
16. Pre-requisites for High Availability
• Optional setups:
• Fast Start Failover Data guard
Setup(Requires additional hardware)
• Load Balancer NOT REQUIRED –
Although the documentation disagrees
• RAC DB’s– Great if you can afford it.
Slide 16
17. Requirements Standby OMS
• Machine Required for Standby OMS/DB
(Linux, Sparc, HPUX, Microsoft)
• Plugins need to be the same version
• Exact same directory structure
• Load Balancer (optional)
• NFS Drive …. Slide 17
18. Configuration Steps:
1.Install Primary OMS / DB
2.Add targets to Primary
3.Setup NFS Drive – Software Library
4.Add All Plugins required to Primary OMS
5.Add Dataguard - Standby Database
6.Create Standby OMS
7.Convert Standby Database to RAC (IF Available)
Slide 18
19. Installation Steps: (Silent)
Verify NFS Software Library on Standby Machine
Install a “Software-Only” Installation on Standby OMS
Machine
(Do NOT run the ConfigureGC.sh or ConfigureGC.bat)
Slide 19
20. Installation Steps: (Silent)
• Ensure same Plugins as the source OMS
SOURCE_OMS_pgrid01>l
1 SELECT epv.plugin_id, epv.version, epv.rev_version
2 FROM em_plugin_version epv, em_current_deployed_plugin ecp
3 WHERE epv.plugin_type NOT IN ('BUILT_IN_TARGET_TYPE','INSTALL_HOME')
4 AND ecp.dest_type='2'
5* AND epv.plugin_version_id = ecp.plugin_version_id
SYSMAN_pgrid01>/
PLUGIN_ID VERSION REV_VERSION
-------------------- ------------ -----------
oracle.sysman.mos 12.1.0.2.0 0
oracle.em.satc 12.1.0.1.0 0
oracle.em.smss 12.1.0.2.0 0
oracle.sysman.emct 12.1.0.3.0 0
oracle.sysman.ssa 12.1.0.4.0 0
oracle.sysman.db 12.1.0.2.0 20120804
oracle.sysman.xa 12.1.0.3.0 0
oracle.sysman.emas 12.1.0.3.0 0
oracle.sysman.vt 12.1.0.3.0 0
pythian.mysql.prod 12.1.0.1.2 0
Slide 20
21. Installation Steps: (Silent)
Plug-in Definitions:
PLUGIN_ID DEFINITION
-------------------- -----------------------
oracle.sysman.mos My Oracle Support
oracle.em.satc Apache Tomcat
oracle.em.smss Microsoft SQL Server
oracle.sysman.emct Charge Back
oracle.sysman.ssa Cloud Application
oracle.sysman.db Oracle Database
oracle.sysman.xa Exadata
oracle.sysman.emas Fusion Middleware
oracle.sysman.vt Virtualisation
pythian.mysql.prod MySQL
Slide 21
22. Installation Steps: (Silent)
Plug-ins (Continued)
1. To Install Additional Plug-ins – Manually download the
plug-ins from OTN to Accessible location (i.e., /tmp)
a. Verify that they are the same version as primary OMS
2. http://www.oracle.com/technetwork/oem/grid-
control/downloads/index.html
3. Run $OMS_HOME/sysman/install/PluginInstall.sh –
pluginLocation /tmp - Choose the plugins you want to
install (screen print – next page)
Slide 22
24. Installation Steps: (Silent)
1. Apply all patches that were applied to the first OMS
a. To Check - $OMS_HOME/OPatch/opatch lsinventory
2. Export the Configuration Details from the First OMS
a. $OMS_HOME/bin/emctl exportconfig oms –dir <Path
Name>
b. Creates bka file – i.e. opf_ADMIN_<DATE>_<TIME>.bka
c. Copy bka file to standby OMS Machine
Slide 24
25. Installation Steps: (Silent)
oracle@pr2prodora01 ~]$ cd /u01/app/oracle/OracleHomes2/oms/bin/
[oracle@pr2prodora01 bin]$ ./emctl exportconfig oms -dir $HOME
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
ExportConfig started...
Machine is Admin Server host. Performing Admin Server backup...
Exporting emoms properties...
Exporting secure properties...
Export has determined that the OMS is not fronted
by an SLB. The local hostname was NOT exported.
The exported data can be imported on any host but
resecure of all agents will be required. Please
see the EM Advanced Configuration Guide for more
details.
Exporting configuration for pluggable modules...
Preparing archive file...
Backup has been written to file: /home/oracle/opf_ADMIN_20130125_101524.bka
The export file contains sensitive data.
Please ensure that it is kept secure.
ExportConfig completed successfully!
[oracle@pr2prodora01 bin]$ cd $HOME
[oracle@pr2prodora01 ~]$ scp opf_ADMIN_20130125_101524.bka oracle@aa2prodora11.jstor.org:/home/oracle
oracle@aa2prodora11.jstor.org's password:
opf_ADMIN_20130125_101524.bka 100% 45MB 5.0MB/s 00:09
[oracle@pr2prodora01 ~]$
Slide 25
26. Installation Steps: (Silent)
OMSCA Breakdown on Standby database using the
backup file created
• Uses the backup file just generated = bka file
• Admin Server HTTPS Port = 7101
• Managed Server Port=7202
• Managed Server HTTPS Port=7301
• EM Note Manager Port = 7403
• EM Upload Port = 4889
• EM Upload HTTPS Port=4900
• EM Console Port = 7788
• EM Console HTTPS Port= 7799
• Config_home = /u01/oracle/product/OracleHomes2/gc_inst
• EM_INSTANCE_HOST aa2prodora11.jstor.org (Standby machine)
Slide 26
27. Installation Steps: (Silent)
Of all these ports the following are relevant to the load
Balancer:
•EM Upload HTTP Port: 4889
•EM UPLOAD HTTPS Port: 4900
•EM Central Console HTTP Port: 7788
•EM Central Console HTTPS Port: 7799
Slide 27
28. Installation Steps: (Silent)
• Run OMSCA on Standby database using the backup
file created
<OMS_HOME>/bin/omsca recover -ms -backup_file
/opt/oracle/product/backup/opf_ADMIN_20120504_031016.bka
-AS_HTTPS_PORT 7101 -MSPORT 7202 -MS_HTTPS_PORT 7301
-EM_NODEMGR_PORT 7403 -EM_ UPLOAD_PORT 4889
-EM_UPLOAD_HTTPS_PORT 4900 -EM_CONSOLE_PORT 7788 -EM_
CONSOLE_HTTPS_PORT 7799 -config_home
/opt/oracle/product/omsmdw/gc_inst -EM_INSTANCE_HOST
example.com
!!! Many errors at this stage, due Failed attempts
Slide 28
29. Installation Issues:
MSEVERE: OMSCA-ERR:Post "Deploy and Repos
Setup" operations failed
Check the following items :
-- Check RAM on Standby Machine – needs 6GB – to Install
-- If using 32 Bit Linux –
Modify – create_domain.py (under omsca/scripts/wls
directory)
Change - -Xmx1524m to –Xmx1324m
Change –XX:MaxPermsize=612M to –X:MaxPermsize=412M
-- Check for Lock in repository
Slide 29
30. Installation Issues
• Check for lock in the repository
• Due to multiple tries there was a lock in the
repository during the plugin deployment stage.
• select host_url, oms_id, status, is_first_oms,
em_release_version from gc_oms_info;
• select mp_guid,file_id,file_name,file_type from
mgmt_mp_files;
• select * from em_plugin_lock_track;
Slide 30
31. Installation Issues:
• First Issue – wrong version of the database plugin
• I upgraded the database plugin but didn’t include it
in the standby installation
• Copied latest “DB” plugin and reran
pluginInstall.sh
Slide 31
32. Installation Issues:
• 2nd Issue – Plugin Issue and multiple attempts caused
lock in EM database (SYSMAN)
• Removed record from lock table
em_plugin_lock_track
• Ran $OMS_HOME/bin - ./pluginca –action deploy
–isFirstOMS false –plugins
• Started OMS
Slide 32
33. Final Installation Steps
If you haven’t installed the agent on the standby
host:
$<AGENT_HOME>/sysman/install/agentDeploy.sh
AGENT_BASE_DIR=<middleware_ home>/agent
OMS_HOST=<second_oms_host_name> EM_UPLOAD_PORT=<second_oms_
port> AGENT_REGISTRATION_PASSWORD=<password> -configOnly
Note: If you have Server Load Balancer (SLB) configured, then directly
enter the host name and port number of the SLB. If SLB is not
configured, then enter the secure upload port of the first OMS.
Slide 33
34. Final Installation Steps
After agent is installed and deployed
• Deploy the required plug-ins on the management agent
Secure the agent - $AGENT_HOME/bin/emctl secure
add_trust_cert_to_jks (If applicable)
In the new OMS Grid control (standby) – discover the Oracle Weblogic
services
• Targets, Middleware, click EMGC_DOMAIN
• From EMGC_DOMAIN home page
• Select Farm Menu, Refresh Weblogic Domain, click continue till 2nd
management server is discovered on the standby OMS Host.
Slide 34
36. What We Covered
• High Available architecture options
• Silent Standby OMS installations and
gotchas
Slide 50
37. References
Deploying a Highly Available EM12C Cloud Control – An Oracle
White Paper – May 2012 - # 1631423.pdf
www.dbakevlar.com – Kellyn Pot’Vin’s blog contains many great
ideas about EM12c
e24089.pdf – Cloud Control Advanced Installation and
Configuration Guide (section 5 - installing Add'l OMS in silent
mode)
39. References
www.pythian.com – MySQL plug-in and instructions
oemgc.wordpress.com – Rob Zoeteweij – Good information on
Incident management and ITL, including a blog
www.slideshare.net/Enkitec/em12c-monitoring-metric-
extensions-and-performance-pages#btnPrevious – Em12C
Monitoring, Metric Extensions and Performance pages – By
Kellyn Pot’Vin – Enkitec
EM12C Log Cleanup – Script to clean up logs and information
41. Thank You for Attending
Please fill out your Evaluations
www.ioug.org/eval
Cloud Control Recipes for EM12C
Session # 941
Mike Gangler
•Email: michael.gangler@ithaka.org
• Twitter: @mjgangler
• Blog: mjgangler.wordpress.com
Slide 55
Editor's Notes
----- Meeting Notes (3/12/13 19:40) ----- Ask Questions regarding using CLoud control How many used 11g Grid Control? 10G 9 ? Earlier - ?
I have worked for Oracle a long time since version 4 and 5.1c and have seen the growth of Enterprise manager / Grid Control / Cloud control since the early days of the “Work Group Stations”, to Version 10 and now Version 12c. I never used Version 11g of Grid Control and havent heard good things about that
Used 9i or less (i.e., workgroup) ? 10g Grid Control ? 11G Grid Control ? 12G Grid Control ? Additional OMS ?
Used 9i or less (i.e., workgroup) ? 10g Grid Control ? 11G Grid Control ? 12G Grid Control ? Additional OMS ?
Used 9i or less (i.e., workgroup) ? 10g Grid Control ? 11G Grid Control ? 12G Grid Control ? Additional OMS ?
Level 3 utilizes multiple Mangement Services (Primary and Secondary) accessed through a local load-balncer and a databases that uses RAC. Also they recommend that the database and OMS are on the same host or close proximity to reduce latency. Level 3 provides continuous availabilyt when a database host or OMS host fails. Furthermore, to inenhance the the availability, we added a Dataguard setup. Note – Level 3 doesn’ t protect against site failure. Level 4 would put the Standby OMS and dataguard on athnterr rac site. ADDING RAC Gives you additioinal protection – Level 4 ----- Meeting Notes (3/12/13 15:45) ----- Level 1 is all contained on 1 site/machine and if that machine fails your OMS is not monitoring. Level 2 is is the same as level 1 but you have 2 OMS setup in an active / Passive mode - failover is via VIP or Load balancer
Level 3 utilizes multiple Mangement Services (Primary and Secondary) accessed through a local load-balncer and a databases that uses RAC. Also they recommend that the database and OMS are on the same host or close proximity to reduce latency. Level 3 provides continuous availabilyt when a database host or OMS host fails. Furthermore, to inenhance the the availability, we added a Dataguard setup. Note – Level 3 doesn’ t protect against site failure. Level 4 would put the Standby OMS and dataguard on athnterr rac site. ADDING RAC Gives you additioinal protection – Level 4 ----- Meeting Notes (3/12/13 15:45) ----- THis level 3 definition is the standard definition, in our situation i created a 3.5 level without RAC or load balancer.
Level 3 utilizes multiple Mangement Services (Primary and Secondary) accessed through a local load-balncer and a databases that uses RAC. Also they recommend that the database and OMS are on the same host or close proximity to reduce latency. Level 3 provides continuous availabilyt when a database host or OMS host fails. Furthermore, to inenhance the the availability, we added a Dataguard setup. Note – Level 3 doesn’ t protect against site failure. Level 4 would put the Standby OMS and dataguard on athnterr rac site. ADDING RAC Gives you additioinal protection – Level 4 ----- Meeting Notes (3/12/13 15:45) ----- you could have multiple Standby OMS in active mode with the load balancers
One of the other reasons for setting this up is to consolated our job scheduler and jobs (Out of cron). This would help create a high available job scheduler as well. Level 3 utilizes multiple Mangement Services (Primary and Secondary) accessed through a local load-balncer and a databases that uses RAC. Also they recommend that the database and OMS are on the same host or close proximity to reduce latency. Level 3 provides continuous availabilyt when a database host or OMS host fails. Furthermore, to inenhance the the availability, we added a Dataguard setup. Note – Level 3 doesn’ t protect against site failure. Level 4 would put the Standby OMS and dataguard on athnterr rac site. ADDING RAC Gives you additioinal protection – Level 4 ----- Meeting Notes (3/12/13 15:45) ----- The dev/qa environments allows you to test patches, plugins, etc prior to going to produciton ----- Meeting Notes (3/12/13 15:48) ----- This will require at least 5 machines - 1 for the oms, 1 for standby OMS, 1 for dg observer, 1 standby db, and 1 primary db.
Produciton – 24 hours a day – 7 days a week Development/QA – 14 hours a day – 6:00am – 8:00pm – Monday – Friday SqlServer Databases – 14 Hours a day – 6:00am – 8:00pm – Monday – Friday MySql Databases – 14 hours a day – 6:00am – 8:00pm – Monday - Friday
NOTE – The diagram should have a DNS in between the Datacenters and the OMS and 1 between the databases. ----- Meeting Notes (3/12/13 15:55) ----- This presentation wont be focusing on the data guard, NFS Shared library or the load balancer, but on setting up the standby OMS, which currently is very buggy and hopefully will be fixed in the next release.
----- Meeting Notes (3/12/13 15:55) ----- NFS Shared is a
----- Meeting Notes (3/12/13 15:55) ----- additional hardware is talking about the dg observer.
If you want to put your Primary and Standby DB’s on separate machines, that will also be required. This application is not a friendly application for shariing (i.e. database farms) and will use the entire instance for users, etc.
----- Meeting Notes (3/12/13 15:55) ----- Silent install is currently required due to Buggy GUI interface ----- Meeting Notes (3/12/13 20:58) ----- after root.sh DO NOT RUN - caps
----- Meeting Notes (3/12/13 16:04) ----- Exports the OMS and security properties as well - this output is saying that the Local host name was not export
The command is OMSCA in the OMS HOME/bin directory. This is all the settings
----- Meeting Notes (3/12/13 20:58) ----- Open paranethesis
----- Meeting Notes (3/12/13 20:58) ----- havent
----- Meeting Notes (3/12/13 20:58) ----- add --> add space after comma
Events - The Entity Could be a target, config file, job, etc. - Example is database down, job failed, host cpu Percentage exceeded. Incident – Might be as simple as the relation with a singe event – Alert Log ora-0600 error or more completex as an server running out of resources (CPU, memory, io
Example of our production database incident rule set Broken Down into : DB Critical Events, DB Up/Down event and Streams Only rule sets
These alerts / rules will automaticallly clear out events based on the
I found most of them didn ’t pertain to our environment or making my own provided better alerts and notifications.
I found most of them didn ’t pertain to our environment or making my own provided better alerts and notifications. The only ones we use are Targets Unreachable and target error. Note most are based on Events
Example of some of the criteria and Actions for each out of the box rule sets Service Level Agreements – Any / All Service Level agreement events that are critical will create an incident but NOT EMAIL Targets Unreachible – Any Host or Agent event that indicate unreachible – Create An Incident and send an Email to the sysman users Targets Down – Any Target availability Event will just create an incident – NOT EMAIL – (I created my own for hosts, databases, etc) Target Error - Any Errors regarding a target availability Event – will send an email to the sysman users and create an incident Clear ADP > 7 days- if an event has been open for 7 days and not associated with an incident - clear
This is an example of our Host Critical rule set: If any host metric Alerts match the following – Severity in Critical, fatal and category in Availability and capacity) Do the following : Email to the syssman users Create an incident – assign to sysman user Update incident (If duplicate) and set it to sysman And then clear the event if no longer an issue.
This is an example of our database rule set: We have 3 rules : - If any host metric Alerts match the following – Severity in Critical, fatal and category in Availability and capacity) Do the following : Email to the syssman users Create an incident – assign to sysman user Update incident (If duplicate) and set it to sysman And then clear the event if no longer an issue.
This is an example of our database rule set: We have 3 rules : - If any host metric Alerts match the following – Severity in Critical, fatal and category in Availability and capacity) Do the following : Email to the syssman users Create an incident – assign to sysman user Update incident (If duplicate) and set it to sysman And then clear the event if no longer an issue.
This is an example of our database rule set: We have 3 rules : - If any host metric Alerts match the following – Severity in Critical, fatal and category in Availability and capacity) Do the following : Email to the syssman users Create an incident – assign to sysman user Update incident (If duplicate) and set it to sysman And then clear the event if no longer an issue.