Oracle RAC 12c Collaborate Best Practices - IOUG 2014 version

Oracle Real Application
Clusters (RAC) 12c
Collaborate Best Practices
Markus Michalewicz
Director of Product Management
Oracle Real Application Clusters (RAC)
@OracleRACpm
http://www.linkedin.com/in/markusmichalewicz
http://www.slideshare.net/MarkusMichalewicz

Safe Harbor Statement
The following is intended to outline our general product direction. It is
intended for information purposes only, and may not be incorporated
into any contract. It is not a commitment to deliver any material, code, or
functionality, and should not be relied upon in making purchasing
decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole
discretion of Oracle.

Agenda
!  Oracle RAC 12c and Standardization
!  Installation of Oracle Grid Infrastructure 12c
!  Installation of the Oracle Database Home
!  Oracle RAC Database Creation using DBCA
!  Post-install Steps and Configuration
!  Summary and Q&A
!  Appendices A-C

Oracle RAC – Family of Solutions
For more information see:
http://www.slideshare.net/MarkusMichalewicz/
http://www.slideshare.net/MarkusMichalewicz/the-oracle-real-application-clusters-rac-family-of-solutions

Standardize on Clusters
Clusters increase flexibility and HA regardless of DB types used later
High Availability
Flexibility
License Costs
Re-configuration
to enable cluster

Standardize on Oracle RAC
For all the flexibility you need
Flexibility
Scalability
Online
upgrade
Online (rolling) Patch and PSU Application

Standardize on Oracle RAC and Oracle Multitenant
The new standard for Oracle Database Consolidation
Consolidation
Agility
More on Fiday, Apr 11th, Session
2548 Oracle Multinenant meets
RAC – Level 3, Lido 3103, 11am

Agenda
!  Appendices A-C

Preparation of the Servers
!  Setup of the Servers include:
–  Setting up network connections
!  Public and private with switch
–  Setting up shared storage for all servers
!  Shared storage required
–  Setting up the OS image – here OL 6.4
!  Suggestion:
–  Use cloning to deploy OS images fastervixencomet
dancerdasher
OS OL 6.4 OS OL 6.4
OS OL 6.4OS OL 6.4
OS image setup and more

Preparation of the Servers
!  Recommendations:
–  Use OL 6.4+ UEK
–  Use the Oracle preinstall package
!  Available for
–  Oracle Database 11g Rel. 2
–  Oracle Database 12c Rel. 1
–  Additional steps to perform after using the
pre-install package for an Oracle RAC cluster:
!  Add a “grid” user & “dba” group
!  Align UIDs / GUIDs across servers
!  Set SELinux to “permissive”, if possible
!  Check large / huge pages settings
Kernel and Oracle preinstall package
[root@dasher ~]# uname –a
Linux dasher 2.6.39-400.17.1.el6uek.x86_64 #1 SMP Fri Feb 22
18:16:18 PST 2013 x86_64 x86_64 x86_64 GNU/Linux
#Get the pre-install package
[root@dasher Desktop]# yum list oracle-*
oracle-rdbms-server-11gR2-preinstall.x86_64 1.0-7.el6 ol6_latest
oracle-rdbms-server-12cR1-preinstall.x86_64 1.0-8.el6 ol6_latest

Install Oracle Grid Infrastructure 12c
What is the default when keeping the preselection in steps 1-4…?
!  GI for a Cluster Installation
!  Standard Cluster
!  Typical Installation

Step 3: Double-check the cluster topology to deploy!

Oracle Flex Cluster
The new standard for resource management in the cluster
Hub Nodes
Leaf Nodes
!  Oracle Flex Cluster:
–  The new standard for flexible
resource management in the cluster
!  Use it as demand requires it
!  Inherits all Flex ASM HA capabilities
!  Will support more use cases in future
–  Efficient High Availability and
resource management across tiers
!  Lightweight cluster stack on leaf nodes
–  Benefit from virtualization and
physical deployments in one cluster

For now, “Standard Cluster” is the default given the preselection
!  Oracle Flex Cluster
–  Can consist of only HUB-nodes
!  If a Leaf-node has access to shared storage,
it can perform a role change to a HUB-node
–  A Standard Cluster can be converted to a
Flex Cluster at any time after the installation
!  Once a cluster is configured as a Flex Cluster
this configuration is irreversible without a re-
configuration of the cluster.
!  Recommendation:
–  An all-HUB-node Flex Cluster behaves the
same as a Standard Cluster using Flex ASM.
–  Either can be used, but if in doubt, you can
choose standard cluster and convert later.
More Information in Appendix A

Step 4: Do not accept the default – choose “Advanced Installation”
!  Typical Installation
–  Does not provide an option to use “Flex ASM”
–  Does not provide an option to create the “Mgmt
DB / Grid Infrastructure Management Repository”
!  Enhancement Requests for later
releases have been filed already
!  Advanced Option
–  Recommended for all configurations

“Advanced Installation” – SCAN & GNS requirement
–  SCAN remains to be required for the cluster
–  Choose cluster-specific unique SCAN names
–  OUI will try to enforce it as much as possible
!  http://www.oracle.com/technetwork/products/
clustering/overview/scan-129069.pdf
–  Choose “Configure GNS” if:
!  You want to use dynamic IP management
(DHCP assigned / IPv6 Stateless Address
Autoconfiguration) for client connectivity
–  In this case, GNS requirements remain as
with previous version for standard cluster
!  You chose to install an Oracle Flex Cluster

“Advanced Installation” – Network Interface Usage
–  Use “Redundant Interconnect Usage” feature
!  Bonding can be used underneath if required
!  MOS notes explain further configuration – e.g.:
–  MOS note 1210883.1 - 11gR2 Grid
Infrastructure Redundant Interconnect
and ora.cluster_interconnect.haip
• It is recommended to align the
ASM network with the private
interconnect.
• Using different subnets for the
private interconnect is required.

“Advanced Installation” – Grid Infrastructure Management Repository
!  Grid Infrastructure Management Repository (GIMR)
–  A Single Instance Oracle Database 12c
–  It will be installed on one of the nodes
!  It is managed as a failover database
–  It contains the Cluster Health Monitor (CHM) data
–  It is stored in the first ASM disk group created
–  As part of the configuration, Quality of Service
Management (QoS) required resources are
added to the cluster configuration (e.g. OC4J)
!  Recommendation: configure the GIMR!

“Advanced Installation” – Storage Options
!  For Standard Cluster Installations
three storage options are available:
1.  Standard ASM
!  ASM configuration as known
from previous releases
2.  Oracle Flex ASM
!  Recommended option available
with Oracle Grid Infrastructure 12c
–  Flex Cluster requires Flex ASM
3.  Shared File System
!  Alternative

Oracle ASM 12c – Overview
Oracle ASM 12c Standard Deployment
ASM Cluster Pool of Storage
Disk Group BDisk Group AShared Disk
Groups
Wide File Striping
One to One
Mapping of ASM
Instances to
Servers
ASM Instance
Database Instance
ASM Disk
RAC Cluster
Node4Node3Node2Node1 Node5ASM ASM ASM ASM ASM
ASM Instance
Database Instance
DBA DBA DBCDBB DBBDBB

Introducing Oracle Flex ASM
Removal of One to One Mapping and HA
ASM Cluster Pool of Storage
Disk Group BDisk Group AShared Disk
Groups
Wide File Striping
Databases share
ASM instances
ASM Instance
Database Instance
ASM Disk
RAC Cluster
Node5Node4Node3Node2Node1
Node5
runs as
ASM Client
to Node4
Node1
runs as
ASM Client
to Node2
Node1
runs as
ASM Client
to Node4
Node2
runs as
ASM Client
to Node3
ASM ASM ASM
ASM Instance
DBA DBA DBCDBB DBBDBB

Use Oracle Flex ASM for storage management
–  Use Oracle Flex ASM for storage
!  For better HA of database instances
!  Reserve per-node spare memory
[GRID]> ps -ef |grep pmon
grid 3822 1 0 08:05 ? 00:00:00 asm_pmon_+ASM1
grid 4865 1 0 08:06 ? 00:00:00 mdb_pmon_-MGMTDB
grid 9592 1 0 08:09 ? 00:00:00 apx_pmon_+APX1
grid 19114 19003 0 09:22 pts/0 00:00:00 grep pmon
More Information in Appendix B

“Advanced Installation” – Create ASM Disk Group
–  Create Infrastructure Disk Group (DG) first
!  Use it to host OCR / Voting Disk
–  Choose Redundancy accordingly
–  Normal(+) redundancy is recommended
–  Create Data and FRA disk groups
post installation and as required.
!  Optional: Use it to host shared DB homes
by creating an ACFS file system on top of it
–  http://www.oracle.com/technetwork/products/cloud-storage/
oop-patching-of-acfs-shared-oh-1885763.pdf
–  ADVM volumes can be of lower
redundancy than the underlying DG

“Advanced Installation” – Prerequisite Checks: Use FixUp Scripts!

New Memory Requirements
Just something to consider
[GRID]> srvctl status asm
ASM is running on comet,dasher,dancer
[GRID]> srvctl status mgmtdb
Database is enabled
Instance -MGMTDB is running on node dasher
grid 3227 1 0 13:54 ? 00:00:02 asm_pmon_+ASM1
grid 7364 1 0 13:57 ? 00:00:01 apx_pmon_+APX1
oracle 7917 1 0 13:57 ? 00:00:02 ora_pmon_raccdb1_4
[RAC]> df -h
Filesystem Size Used Avail Use% Mounted on
...
tmpfs 1.7G 1.3G 475M 73% /dev/shm
...
Physical memory requirements
with Oracle RAC 12c: 4GB

Agenda
!  Appendices A-C

Install the Oracle Database Home
Nothing special – just a quick walk through
Not the
default…

Install the Oracle Database Home
A few things to point out
ACFS based
shared home …

Before Creating Databases
Create Server Pools and additional disk groups as required
[GRID]$ srvctl add srvpool –serverpool backoffice
–min 1 –max 2 –i 1
[GRID]$ srvctl add srvpool –serverpool frontoffice
–min 1 –max 4 –i 2
Create additional disk
groups as required.

Agenda
!  Appendices A-C

Create an Oracle Database
What is the default when keeping the preselection in steps 1-4…?
!  Policy-managed
!  Oracle RAC-based
!  Non-CDB Database

Create an Oracle RAC Database
A few tips and tricks…
Create an empty
multitenant CDB.

Create an Oracle RAC Database
A few tips and tricks…
This is the default…,
but not recommended
–  Use a different disk group than the “database file
location”-disk group as the Fast Recovery Area.
–  Automatic Storage Management (ASM)
recommendation remains unchanged
!  Use it for any current production database
!  New in Oracle Database 12c:
ACFS can be used to host database files
–  Use ACFS for database files, if you want to
!  manage “all data” on a file system
!  use advanced functionality

Milestone
What has been installed and configured so far?
vixen srvpool: frontofficecomet srvpool: backoffice
dancer srvpool: backofficedasher srvpool: frontoffice
OS OL 6.4 OS OL 6.4
OS OL 6.4OS OL 6.4
Oracle GI for a cluster Oracle GI for a cluster
Oracle GI for a clusterOracle GI for a cluster
Oracle RAC Oracle RAC
Oracle RACOracle RAC
raccdb1
raccdb1_2
raccdb1_1
[RAC]> srvctl status database -d raccdb1
Instance raccdb1_1 is running on node comet
Instance raccdb1_2 is running on node dancer
[RAC]>srvctl status srvpool -serverpool backoffice -detail
Server pool name: backoffice
Active servers count: 2
Active server names: comet,dancer
NAME=comet STATE=ONLINE
NAME=dancer STATE=ONLINE
[RAC]> srvctl status srvpool
Server pool name: Free
Server pool name: Generic
Server pool name: frontoffice

Create Pluggable Databases
Create HR PDB on running instances
raccdb1
raccdb1_2
raccdb1_1
Create a PDB on a node
that hosts a multintenant
CDB instance

Agenda
!  Appendices A-C

Post-Creation Steps
Create services to balance the workload
raccdb1
raccdb1_2
raccdb1_1 [RAC]> srvctl add service -d raccdb1 -s hrsvc -pdb hr -
serverpool backoffice
[RAC]> srvctl add service -d raccdb1 -s crmsvc -pdb crm -
serverpool frontoffice
!  Use Dynamic Database Services
–  For easier management of multitenant CDBs
–  For better workload management in the cluster
!  Start services after creation and as required
Details on FRI, Apr 11th, Session
2548 Oracle Multinenant meets
RAC – Level 3, Lido 3103, 11am
More Information in Appendix C

Agenda
–  Testing
–  Monitoring &Analysis
!  Appendices A-C

Engineered Systems
Optimized and already tested
Oracle Exadata
Database Machine
Oracle Database
Appliance
Quarter Rack
Half Rack
Full Rack
• IB Interconnect
• Low Overhead IPC Prot.
Preconfigured

Oracle Real Application Testing
Gather and deploy workload on any database type…
ERP
CRM
DW

Standardize on Oracle RAC (One Node)
Here: to simplify Oracle RAC testing …
Flexibility
Scalability
Re-configuration
to enable cluster
Online
upgrade

Standardize on Oracle RAC (One Node)
Oracle RAC testing made simple:
1.  Standardize on Oracle RAC (One Node)
2.  Run tests on Oracle RAC One Node instance
–  RAC One Node behaves basically like a single instance
3.  Online Upgrade to Oracle RAC for testing only
4.  Run tests on Oracle RAC
5.  Decide whether the application provides service
goals when load balanced over multiple active
instances
–  Identify bottlenecks and tuning actions as required
To simplify Oracle RAC testing
vixencomet
RON RON_2
Online
upgrade
RON_1
AWR
---
ASH
AWR
---
ASH
Decide
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC

Agenda
–  Testing
–  Monitoring &Analysis
!  Appendices A-C

Post-Install Steps
Check your environment regularly using CVU and ORAcheck
#CVU is run regularly by default
[GRID]> srvctl status cvu
CVU is enabled and running on node comet
!  CVU is run regularly as part of the GI stack
!  Use ORAcheck (formerly RACcheck)
the Oracle RAC Configuration Audit Tool
–  For details see MOS note 1268927.1
!  Checks the Oracle Database:
–  Standalone Database
–  Grid Infrastructure & RAC
–  Maximum Availability Architecture (MAA) Validation
–  Upgrade Readiness Validation
–  Golden Gate
!  E-Business Suite
!  Oracle SUN Systems

Monitoring Essentials
dasher dancer
vixencomet
Use Cluster Health Monitor for OS (CHM/OS)
1.  First failure diagnostics: Collects as much
system metrics and data as feasible
2.  Collects system metrics on all
nodes in the cluster synchronized
3.  Persistent storage of all
collected data in a database
4.  Integrated with Oracle Grid Infrastructure
starting with 11.2.0.2 (12c: uses GIMR)
5.  Robust collection in situations
of resource starvation
6.  Data dump or visualization
LOGGERD
sysmond
sysmond
Oracle GI Oracle GI
Oracle GI Oracle GI
sysmond
sysmond
sysmond
LOGGERD

Log Collection Essentials
dasher dancer
vixencomet
Runtime and Emergency Data Collections
!  Save database statistics
–  For baselines and reference: export AWR
!  For diagnostics:
–  Always get AWR and ASH reports
!  For emergencies and “last gasp”
–  get ASH dump (e.g. oradebug ashdunp level 0)
!  CHM/OS
–  oclumon dumpnodeview -n pecdb08 pecdb09 -s
"2011-09-30 15:00:00" -e "2011-09-30 15:30:00" -v
–  oclumom –manage –report <ret. time in secs>
!  CSS logs from all nodes
–  Node and instance evictions
–  Interconnect issues (e.g. gc blocks lots )
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
LOGs
LOGs
LOGs
LOGs

dasher
comet
Use Trace File Analyzer (TFA) to collect Data
!  Goals
–  Improved comprehensive first
failure diagnostics collection
–  Efficient collection, packaging
and transfer of data for customers
–  Reduce round trips between
customers and Oracle
–  Supports 10.2, 11.1, 11.2 and above
–  Included in the 11.2.0.4
patch set and future versions
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
LOGs
LOGs
TFA

dasher
comet
!  Approach
–  Collect for all relevant components
(OS, Grid Infrastructure, ASM, RDBMS)
–  One command to collect all information
–  Prune large files based on temporal criteria
–  Collect time relevant IPS (incident)
packages on RAC nodes
–  Collect time relevant CHM/OS,
OSWatcher data on all Oracle RAC nodes
–  On-demand (default) and
Event Driven diagnostic collections
!  TFA Collector – MOS note ID 1513912.1
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
LOGs
LOGs
TFA

dasher
comet
!  TFA Version 3.2 Updates (07Apr14):
1.  First version of the Log Analyzer.
2.  Addition of –notrim for diagnostic collections.
3.  Support for TFA on zLinux.
4.  Collection of ExaWatcher data.
5.  Sundiag collection
6.  Aging of TFA log files restricting maximum size.
7.  Multiple performance enhancements
8.  Critical Bug fixes
!  TFA Collector – MOS note ID 1513912.1
Oracle GI
Oracle RAC
Oracle GI
Oracle RAC
LOGs
LOGs
TFA

TFA – Usage Example
Sending diagcollect request to host : myhost2
Getting list of files satisfying time range [Tue Sep 03 14:17:43 PDT 2013, Tue Sep 03 18:17:43 PDT 2013]
myhost1: Zipping File: /opt/oracle/oak/oswbb/archive/oswiostat/myhost1_iostat_13.09.03.1500.dat.gz
myhost1: Zipping File: /u01/app/oracle/diag/rdbms/bill/bill1/trace/alert_bill1.log
Trimming file : /u01/app/oracle/diag/rdbms/bill/bill1/trace/alert_bill1.log with original file size : 109kB
myhost1: Zipping File: /opt/oracle/oak/oswbb/archive/oswtop/myhost1_top_13.09.03.1500.dat.gz
myhost1: Zipping File: /opt/oracle/oak/log/myhost1/oak/oakd.log
Trimming file : /opt/oracle/oak/log/myhost1/oak/oakd.log with original file size : 9.2MB
myhost1: Zipping File: /u01/app/11.2.0.3/grid/log/myhost1/gipcd/gipcd.log
myhost1: Zipping File: /u01/app/11.2.0.3/grid/log/myhost1/agent/ohasd/oraagent_grid/oraagent_grid.log
Trimming file : /u01/app/11.2.0.3/grid/log/myhost1/agent/ohasd/oraagent_grid/oraagent_grid.log with original filesize 4.3MB
myhost1: Zipping File: /var/log/messages
…Truncated for brevity
myhost1: Zipping File: /opt/oracle/oak/oswbb/archive/oswslabinfo/myhost1_slabinfo_13.09.03.1800.dat
Collecting ADR incident files...
Total Number of Files checked : 10543
Total Size of all Files Checked : 3.9GB
Number of files containing required range : 68
Total Size of Files containing required range : 129MB
Number of files trimmed : 10
Total Size of data prior to zip : 144MB
Saved 63MB by trimming files
Zip file size : 8.6MB
Total time taken : 47s.
Logs are collected to:
/opt/oracle/tfa/tfa_home/repository/collection_Tue_Sep_3_18_17_24_PDT_2013_node_all/myhost1.tfa_Tue_Sep_3_18_17_24_PDT_2013.zip
/opt/oracle/tfa/tfa_home/repository/collection_Tue_Sep_3_18_17_24_PDT_2013_node_all/myhost2.tfa_Tue_Sep_3_18_17_24_PDT_2013.zip
OS Watcher files
Pruning
47 seconds! – 1 command, 2 nodes, 4 databases, ASM, Clusterware, OS
Relevant files only
$ sudo ./tfactl diagcollect
144MB pruned and compressed down to 8.6MB
One simple command
ADR Incident files

Analysis Essentials
Time Series Analysis – AWR Reports are not enough due to average…
Occurrence of an interconnect
problem ~18 secs avg !!!!
!  Solution
–  ASH reports
Every
1 hour
or
out-of-space
AWR
Circular buffer
in SGA
(2MB per CPU)
DBA_HIST_ACTIVE_SESS_HISTORYV$ACTIVE_SESSION_HISTORY
Session
state
objects
MMON Lite
(MMNL)
V$SESSION
V$SESSION_WAIT
Write 1
out of 10
samples
Variable
length rows
Direct path
inserts

Post-Install Steps
Monitor best practices notes and announcements
!  ORAcheck - Oracle Configuration Audit Tool
–  For details see MOS note 1268927.1
!  Best practices My Oracle Support (MOS) notes:
–  MOS note 810394.1 - RAC and Oracle Clusterware Best Practices and Starter Kit (Platform Independent)
!  MOS note 811306.1 - RAC and Oracle Clusterware Best Practices and Starter Kit (Linux)
!  MOS note 811280.1 - RAC and Oracle Clusterware Best Practices and Starter Kit (Solaris)
–  Note: Engineered Systems with the same OS will have additional recommendations
!  MOS note 811271.1 - RAC and Oracle Clusterware Best Practices and Starter Kit (Windows)
!  MOS note 811303.1 - RAC and Oracle Clusterware Best Practices and Starter Kit (HP-UX)
!  MOS note 811293.1 - RAC and Oracle Clusterware Best Practices and Starter Kit (AIX)
!  MOS note 1452965.2 –
–  Information Center: Oracle Scalability Grid Infrastructure / Clusterware and Real Application Clusters (RAC)

Agenda
!  Appendices A-C

More Information – Extended RAC and More
!  Presentation: Oracle RAC on Extended Distance Clusters
–  http://www.slideshare.net/MarkusMichalewicz/oracle-rac-on-extended-distance-clusters
!  WP: Oracle Real Application Clusters on Extended Distance Clusters
–  http://www.oracle.com/technetwork/database/options/clustering/overview/extendedracversion11-435972.pdf
!  WP: Using standard NFS to support a third voting file for extended cluster configurations
–  http://www.oracle.com/technetwork/database/database-technologies/clusterware/overview/grid-infra-thirdvoteonnfs-131158.pdf
!  Presentation: Oracle Clusterware Node Management and Voting Disks
–  http://www.slideshare.net/MarkusMichalewicz/oracle-clusterware-node-management-and-voting-disks
!  Presentation: Oracle Database with RAC 12c HA BP TCOUG
–  http://www.slideshare.net/MarkusMichalewicz/oracle-database-with-rac-12c-ha-bp-tcoug

Questions
and
Answers

Agenda
!  Appendices A-C

Appendix A:
Converting to an Oracle
Flex Cluster with Node
Conversion Example

Oracle Flex Cluster
Some configuration details
!  Oracle Flex Cluster
–  Will be the basis for more uses cases
in future releases (for resource mgmt)
–  Can consist of only HUB-nodes
–  If a Leaf-node has access to shared storage,
it can perform a role change to a HUB-node
–  A Standard Cluster can be converted to a
Flex Cluster at any time after the installation
–  Once a cluster is configured as a Flex Cluster
this configuration is irreversible without a re-
configuration of the cluster.
!  More Information:
–  http://docs.oracle.com/cd/E16655_01/rac.121/e17886/
bigcluster.htm#CWADD92560
[GRID]> crsctl get cluster mode config
Cluster is configured as "standard"
[GRID]> crsctl get cluster mode status
Cluster is running in "standard" mode
[GRID]> olsnodes -s -a -t
dasher Active Hub Unpinned
vixen Active Hub Unpinned
dancer Active Hub Unpinned
comet Active Hub Unpinned
•  Not a Flex Cluster yet.
•  An all-Hub Flex Cluster is
basically a Standard Cluster

Oracle Flex Cluster
Converting to Flex Cluster
bigcluster.htm#CHDJGBII
!  Flex Cluster implementations require GNS
!  Converting a Flex Cluster requires a cluster restart
–  Use “crsctl stop crs” followed by “crsctl start crs” as root
–  Hence the difference between “status” and “config”
–  Nodes can be restarted in a rolling fashion:
[GRID]> srvctl config gns
PRKF-1110 : Neither GNS server nor GNS client is configured on
this cluster
[GRID]> more /etc/hosts |grep gns
10.1.1.99 mygns mygns.localdomain
#Change to root
[GRID]> su
Password:
[GRID]> srvctl add gns -vip mygns
[GRID]> srvctl start gns
[GRID]> crsctl set cluster mode flex
Cluster mode set to "flex »
[GRID]> crsctl get cluster mode config
Cluster is configured as "flex"
[GRID]> date ; hostname ;
crsctl get cluster mode status
Sat Sep 14 14:43:45 PDT 2013
dasher
Cluster is running in "flex" mode
[GRID]> date ; hostname ;
crsctl get cluster mode status
Sat Sep 14 14:43:37 PDT 2013
dancer

Oracle Flex Cluster
Standard to Flex Cluster conversion – what has changed?
Oracle GI | HUB Oracle GI | HUB
Oracle GI | HUBOracle GI | HUB
raccdb1
raccdb1_3
raccdb1_4 raccdb1_2
raccdb1_1
Cluster is running in "flex" mode
[GRID]> olsnodes -a -s -t
dasher Active Hub Unpinned
vixen Active Hub Unpinned
dancer Active Hub Unpinned
comet Active Hub Unpinned
Instance raccdb1_3 is running on node vixen
Instance raccdb1_4 is running on node dasher
!  Nothing at first glance…
–  An all-Hub Flex Cluster behaves
like a Standard Cluster
–  When converting from Standard Cluster
all nodes are converted into HUB-nodes

Oracle Flex Cluster
Changing node roles – server vixen becomes a Leaf node
bigcluster.htm#BABHCEAE
[GRID]> hostname
vixen
[GRID]> srvctl config vip -node vixen
VIP exists: network number 1, hosting node vixen
VIP Name: vixen-vip
VIP IPv4 Address: 10.1.1.33
VIP IPv6 Address:
[GRID]> crsctl get node role config
Node 'vixen' configured role is 'hub'
#Change to root …
[GRID]> su
Password:
[GRID]> crsctl set node role leaf
CRS-4408: Node 'vixen' configured role successfully changed; restart
Oracle High Availability Services for new role to take effect.
[GRID]> crsctl stop crs
[GRID]> crsctl start crs
[GRID]> crsctl get node role config
Node 'vixen' configured role is 'leaf'
[GRID]> srvctl config vip -node vixen
VIP exists: network number 1, hosting node vixen
VIP Name: vixen-vip
VIP IPv4 Address: 10.1.1.33
VIP IPv6 Address:

Oracle Flex Cluster
Node role change to Leaf – what has changed?
vixen srvpool: FREEcomet srvpool: backoffice
Oracle GI | LEAFOracle GI | HUB
raccdb1
raccdb1_4 raccdb1_2
raccdb1_1
[RAC]> srvctl status server -servers vixen -detail
Server active pools: Free
Server name: vixen
Server state: ONLINE
Server state details:
!  Server vixen has converted to a Leaf node
!  Leaf nodes do not host DB instances in 12.1
!  Server vixen moved to Server Pool “FREE” as
Server Pool “frontoffice” is a database pool
–  Note: all nodes in this example have access
to shared storage, which means only the node
role changes – storage access is preserved.

Oracle Flex Cluster
Revert role change – server vixen becomes a HUB again
comet srvpool: backoffice
Oracle GI | HUB
Oracle RAC
raccdb1
raccdb1_4 raccdb1_2
raccdb1_1
vixen srvpool: frontoffice
Oracle GI | HUB
Oracle RAC
raccdb1_3
#As root
[GRID]> crsctl set node role hub
CRS-4408: Node 'vixen' configured role successfully changed;
restart Oracle High Availability Services for new role to take
effect.
[GRID]> crsctl stop crs
[GRID]> crsctl start crs

Appendix B:
Oracle Flex ASM –
Details and Test Scenarios

Oracle Flex ASM
Configuration details
!  Flex ASM introduces new local resources:
–  At least one ASM listener
–  One “proxy_advm” (per node)
!  Used for ACFS access to Flex ASM instances
!  Connections from a database instance to an ASM
instance are based on SQLnet using listeners.
–  The listener directs the connection to the least loaded
ASM instance based on the load metric it maintains.
–  The connection details are fetched from CSS global data
–  The ASM instance to which the database instance
connects to is listed in the database alert log:
!  NOTE: ASMB connected to ASM instance
+ASM1 (Flex mode; client id 0x10004)
–  The userid and password supplied are also managed
automatically. They are supplied while establishing the
session, not while connecting.
[GRID]> crsctl stat res -t
-------------------------------------------------------------------------------
Name Target State Server State details
-------------------------------------------------------------------------------
Local Resources
-------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE comet STABLE
ONLINE ONLINE dancer STABLE
ONLINE ONLINE dasher STABLE
ONLINE ONLINE vixen STABLE
...
ora.DATA.dg
OFFLINE OFFLINE vixen STABLE
...
ora.INFRA.dg
OFFLINE OFFLINE vixen STABLE
...
ora.proxy_advm
ONLINE ONLINE vixen STABLE

Oracle Flex ASM
Administering Flex ASM
!  Check ASM mode using asmcmd
!  Check where ASM instances are running using srvctl
!  Check the configuration of ASM using srvctl
!  Modify the ASM configuration using srvctl
!  Cluster ASM listener:
–  Flex ASM can have its own network or overlap with the
private interconnect network used by RAC DB instances
–  It is recommended to configure a full overlap of the private
interconnect and the ASM network in current use cases:
[GRID]> asmcmd
ASMCMD> showclustermode
ASM cluster : Flex mode enabled
[GRID]> srvctl config asm -detail
ASM home: /u01/app/12.1.0/grid
Password file: +INFRA/orapwASM
ASM listener: LISTENER
ASM is enabled.
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM,ASMNET2LSNR_ASM
[GRID]> srvctl modify asm -count all
[GRID]> srvctl start asm
ASM is running on comet,vixen,dasher,dancer

Oracle Flex ASM
Some test scenarios…
!  A failure of an ASM instance is without effect for the
databases using ASM disk groups (here: -MGMTDB)
–  If an ASM instance is running on all nodes (count all)
the ASM instance is restarted upon failure
–  If an ASM instance is running on a subset of nodes
the ASM instance might be restarted on another node
–  The client using the ASM instance may change
!  A failure of an ASM instance is without effect for
the APX (ADVM proxy) running on the same node
!  A failure of the APX will lead to a restart of the database
instance (on the node) that uses an ACFS file system.
!  More information:
–  http://docs.oracle.com/cd/E16655_01/server.121/e17612/
asminst.htm#OSTMG95329
grid 4093 1 0 11:35 ? 00:00:00 asm_pmon_+ASM1
grid 4548 1 0 11:36 ? 00:00:00 apx_pmon_+APX1
[GRID]> kill -9 4093
grid 4548 1 0 11:36 ? 00:00:00 apx_pmon_+APX1
grid 4548 1 0 11:36 ? 00:00:00 apx_pmon_+APX1
grid 5748 1 0 11:39 ? 00:00:00 asm_pmon_+ASM1
grid 5793 19013 0 11:39 pts/1 00:00:00 grep pmon
[GRID]> asmcmd lsct data
DB_Name Status Software_Version Compatible_version Instance_Name Disk_Group
+APX CONNECTED 12.1.0.1.0 12.0.0.0.0 +APX1 DATA
[GRID]> ssh vixen /u01/app/12.1.0/grid/bin/asmcmd lsct data
DB_Name Status Software_Version Compatible_version Instance_Name Disk_Group

Appendix C:
Policy-Managed Databases
and with Oracle RAC 12c

Policy-Managed Databases with Oracle RAC 12c
Database Services
Oracle GI | HUB
Oracle RAC
raccdb1
raccdb1_3
raccdb1_4 raccdb1_2
[GRID]> srvctl status serverpool
Active Servers count: 2
[RAC]> srvctl status service -d raccdb1
Service crmsvc is running on nodes: dasher,vixen
Service hrsvc is running on nodes: comet,dancer
!  Database Services
–  Ensure that workload is hosted in the respective
server pool with the cardinality defined as part of
the policy definition.
Oracle GI | HUB
Oracle RAC
raccdb1_1

Improved HA Management - Startup
raccdb1
[GRID]> srvctl config serverpool
Importance: 10, Min: 2, Max 2
Importance: 0, Min: 0, Max -1
!  Defining a Service Startup Order
–  Ensure services are started in specified groups
and specified order.
–  Leverages Serverpool Min and Importance properties
dasher srvpool: frontoffice
Oracle GI | HUB
Oracle RAC
raccdb1_4
1st
Oracle GI | HUB
Oracle RAC
raccdb1_3
2nd
Oracle GI | HUB
Oracle RAC
raccdb1_1
4th
dancer srvpool: backoffice
Oracle GI | HUB
Oracle RAC
raccdb1_2
3rd

Oracle GI | HUB
Oracle RAC
raccdb1_2
dancer srvpool: frontoffice
Oracle GI | HUB
Oracle RAC
raccdb1_2
Improved HA Management – New Failover Strategy
raccdb1
[GRID]> srvctl status serverpool
Oracle GI | HUB
Oracle RAC
raccdb1_4
Oracle GI | HUB
Oracle RAC
raccdb1_3
Oracle GI | HUB
Oracle RAC
raccdb1_1
!  Servers “Move” to Replace Failed node
–  Protects against cascade failures due to load
–  Ensures workload isolation between pools
–  Less important workloads shutdown transactionally

Oracle GI | HUB
Oracle RAC
raccdb1_2
Oracle GI | HUB
Oracle RAC
raccdb1_2
Improved HA Management – Manage Last Service Standing
raccdb1
[RAC]> srvctl config serverpool
[RAC]> srvctl status service –db
Service crmsrv is running on nodes comet,vixen
Service hrsvc is not running
Oracle GI | HUB
Oracle RAC
raccdb1_4
Oracle GI | HUB
Oracle RAC
raccdb1_3
Oracle GI | HUB
Oracle RAC
raccdb1_1
!  Business Critical Services survive multiple failures
–  Most important pool always gets the servers
–  Services preserved across multiple failures
–  Less important workloads shutdown transactionally
comet srvpool: frontoffice
Oracle GI | HUB
Oracle RAC
raccdb1_1

Improved HA Management – Dynamic Provisioning
Oracle GI | HUB
Oracle RAC
raccdb1
raccdb1_3
raccdb1_4 raccdb1_2
[GRID]> srvctl modify serverpool
–serverpool backoffice –max 1
[RAC]> srvctl config serverpool
[RAC]> srvctl status service –db
Service crmsrv is running on nodes dasher,dancer,vixen
Service hrsvc is not running
Oracle GI | HUB
Oracle RAC
raccdb1_1
!  Add Servers Just-In Time to meet demand
–  Server Pools sized via Min and Max properties
–  Dynamically controlled by QoS Management
–  Planned control via Clusterware or QoSM Policies
Oracle GI | HUB
Oracle RAC
raccdb1_2

Policy-Management with Oracle RAC 12c
Support for Multiple Policies tracking business objectives
–  http://docs.oracle.com/cd/E16655_01/rac.121/e17886/pbmgmt.htm
!  New in Oracle Grid Infrastructure 12c
–  Server Categories
!  Server Categories use server attributes to allow
for an active use of differently sized servers
–  Policy Sets
!  Policy Sets allow for dynamic adjustment
to demand changes in an atomic transaction
Server Categories
•  NAME
•  ACL
•  EXPRESSION
•  …
Server Attributes
•  NAME
•  MEMORY_SIZE!
•  CPU_COUNT
•  CPU_CLOCK_RATE!
•  CPU_HYPERTHREADING!
•  CPU_EQUIVALENCY
• …!
[GRID]> crsctl modify policyset –attr
"LAST_ACTIVATED_POLICY=NightTime“
CRS-2673: Attempting to stop
'ora.raccdb1.crmsvc.svc' on 'comet'
CRS-2673: Attempting to stop
'ora.raccdb1.crmsvc.svc' on 'dancer'
CRS-2677: Stop of 'ora.raccdb1.crmsvc.svc' on
'comet' succeeded
CRS-2673: Attempting to start
'ora.raccdb1.backup.svc' on 'comet'
CRS-2677: Stop of 'ora.raccdb1.crmsvc.svc' on
'dancer' succeeded
CRS-2672: Attempting to start
'ora.raccdb1.hrsvc.svc' on 'dancer‘
…
QoS Management
OR
Oracle Clusterware - CRSCTL

Setting up policy sets – Provision Server Pools and creating a PolicySet
!  Add another server pool “backup”
!  Set up policy set with 3 server pools & 3 policies as follows:
–  DayTime:
!  frontoffice uses three servers (MIN_SIZE=3)
!  backoffice uses one server (MIN_SIZE=1)
!  backup does not run during daytime (MIN_SIZE=0)
–  NightTime:
!  frontoffice uses one server (MIN_SIZE=1)
!  backoffice uses two servers (MIN_SIZE=2)
!  backup uses only one server (MIN_SIZE=1)
–  Weekend:
!  frontoffice uses one server (MIN_SIZE=1)
!  backoffice uses one server (MIN_SIZE=1)
!  backup uses two servers (MIN_SIZE=2)
[GRID]> srvctl add serverpool
–serverpool backup –min 0 –max 2 –importance 20
[GRID] srvctl status serverpool
Server pool name: backup
[GRID]> crsctl modify policyset
-file /u01/app/…/policyset.txt

Using Policy Sets – part 1: Activate a policy
!  Activate the policy for the weekend
[GRID]> crsctl modify policyset
-attr "LAST_ACTIVATED_POLICY=Weekend"
Importance: 0, Min: 0, Max: -1
Category:
Candidate server names:
Server pool name: Generic
Importance: 0, Min: 0, Max: -1
Category:
Importance: 0, Min: 1, Max: 1
Category:
Category:
Category:

Using Policy Sets – part 2: check the result
vixen srvpool: frontofficecomet srvpool: backup
dancer srvpool: backupdasher srvpool: backoffice
raccdb1
raccdb1_3
raccdb1_4 raccdb1_2
raccdb1_1
Service backup is running on nodes: comet,dancer
Service crmsvc is running on nodes: vixen
Service hrsvc is running on nodes: dasher
!  With an activated weekend policy

Adding Server Categories to the picture – part 1
!  Assume you have 2 servers that have better IO
–  Use these servers for backups whenever possible
!  Here comet and dancer have better IO by definition
!  What you need to do:
–  Set up a server category that identifies the servers
–  Add the use of the server category to the server pool
!  Define the server pools that utilize the category
and during which policy activation it shall be used.
!  You need to restart the cluster stack on
the servers that you modify in this fashion
[GRID]> su
Password:
[GRID]> crsctl set server label IOplus
...
#On dancer
[GRID]> crsctl set server label Ioplus
[GRID]> crsctl get server label
CRS-4972: Current SERVER_LABEL parameter value is Ioplus
[GRID]> crsctl status server comet dancer –f
Comet Dancer
NAME=comet
MEMORY_SIZE=3338
CPU_COUNT=1
CPU_CLOCK_RATE=2132
CPU_HYPERTHREADING=0
CPU_EQUIVALENCY=1000
DEPLOYMENT=other
CONFIGURED_CSS_ROLE=hub
RESOURCE_USE_ENABLED=1
SERVER_LABEL=IOplus
...
NAME=dancer
MEMORY_SIZE=3338
CPU_COUNT=1
CPU_CLOCK_RATE=2247
CPU_HYPERTHREADING=0
CPU_EQUIVALENCY=1000
DEPLOYMENT=other
CONFIGURED_CSS_ROLE=hub
RESOURCE_USE_ENABLED=1
SERVER_LABEL=IOplus
...

Adding Server Categories to the picture – part 2
!  What you need to do:
–  Set up a server category that identifies the servers -
done
–  Add the use of the server category to the server pool
!  Define the server pools that utilize the category
and during which policy activation it shall be used
!  Note: Add the server category usage to the
server pool(s) that require it & specify the
policies that should be altered accordingly
–  In this example the backup server pool is the one
that will benefit the most and the policies will only
start this server pool as required.
[GRID]> crsctl add category moreIO
–attr "EXPRESSION='SERVER_LABEL co IOplus'"
[GRID]> crsctl status category moreIO
NAME=moreIO
ACL=owner:grid:rwx,pgrp:oinstall:rwx,other::r--
ACTIVE_CSS_ROLE=hub
EXPRESSION=(SERVER_LABEL co IOplus)
[GRID]> crsctl modify serverpool ora.backup
-attr "SERVER_CATEGORY=moreIO" -all_policies

Adding Server Categories to the picture – comparison
Before After
[GRID]> crsctl status policy DayTime
POLICY
NAME=DayTime
DESCRIPTION=
SERVERPOOL
NAME=Free
...
SERVERPOOL
NAME=ora.backoffice
IMPORTANCE=0
MAX_SIZE=1
MIN_SIZE=1
SERVER_CATEGORY=
SERVER_NAMES=
SERVERPOOL
NAME=ora.backup
IMPORTANCE=0
MAX_SIZE=0
MIN_SIZE=0
SERVER_CATEGORY=
SERVER_NAMES=
SERVERPOOL
NAME=ora.frontoffice
IMPORTANCE=0
MAX_SIZE=3
MIN_SIZE=3
SERVER_CATEGORY=
SERVER_NAMES=
[GRID]> crsctl status policy DayTime
POLICY
NAME=DayTime
DESCRIPTION=
SERVERPOOL
NAME=Free
...
SERVERPOOL
NAME=ora.backoffice
IMPORTANCE=0
MAX_SIZE=1
MIN_SIZE=1
SERVER_CATEGORY=
SERVER_NAMES=
SERVERPOOL
NAME=ora.backup
IMPORTANCE=0
MAX_SIZE=0
MIN_SIZE=0
SERVER_CATEGORY=moreIO
SERVER_NAMES=
SERVERPOOL
NAME=ora.frontoffice
IMPORTANCE=0
MAX_SIZE=3
MIN_SIZE=3
SERVER_CATEGORY=
SERVER_NAMES=

Using Policy Sets and the effect of Server Categories
raccdb1
raccdb1_3
raccdb1_4 raccdb1_2
raccdb1_1
vixen srvpool: backofficecomet srvpool: frontoffice
dancer srvpool: frontofficedasher srvpool: frontoffice
raccdb1
raccdb1_3
raccdb1_4 raccdb1_2
raccdb1_1

[GRID]> crsctl modify policyset –attr
"LAST_ACTIVATED_POLICY=DayTime“
[GRID] srvctl status serverpool
Service backup is not running.
Service crmsvc is running on nodes: dasher,vixen,comet
Service hrsvc is running on nodes: dancer
Using Policy Sets means changing policies on a push of a button
raccdb1
Oracle GI | HUB
Oracle RAC
raccdb1_3
Oracle GI | HUB
Oracle RAC
raccdb1_4
comet srvpool: backup
Oracle GI | HUB
Oracle RAC
raccdb1_1
dancer srvpool: backup
Oracle GI | HUB
Oracle RAC
raccdb1_2
Oracle GI | HUB
Oracle RAC
raccdb1_2
comet srvpool: frontoffice
Oracle GI | HUB
Oracle RAC
raccdb1_1

Using Policy Sets – changing policies on the push of a button is fast …
raccdb1
raccdb1_3
raccdb1_4 raccdb1_2
raccdb1_1
vixen srvpool: backofficecomet srvpool: frontoffice
dancer srvpool: frontofficedasher srvpool: frontoffice
raccdb1
raccdb1_3
raccdb1_4 raccdb1_2
raccdb1_1
[GRID]> date; crsctl modify policyset -attr
"LAST_ACTIVATED_POLICY=DayTime"; date
Mon Sep 16 19:26:42 PDT 2013
CRS-2673: Attempting to stop 'ora.raccdb1.backup.svc' on 'dancer'
CRS-2673: Attempting to stop 'ora.raccdb1.backup.svc' on 'comet'
CRS-2677: Stop of 'ora.raccdb1.backup.svc' on 'dancer' succeeded
CRS-2677: Stop of 'ora.raccdb1.backup.svc' on 'comet' succeeded
CRS-2672: Attempting to start 'ora.raccdb1.crmsvc.svc' on 'dancer'
CRS-2672: Attempting to start 'ora.raccdb1.crmsvc.svc' on 'comet'
CRS-2676: Start of 'ora.raccdb1.crmsvc.svc' on 'dancer' succeeded
CRS-2676: Start of 'ora.raccdb1.crmsvc.svc' on 'comet' succeeded
Mon Sep 16 19:26:43 PDT 2013
" Time to execute: 1 second!

What-If evaluation of policy changes
Service crmsvc is running on nodes: dancer,vixen,dasher
Service hrsvc is running on nodes: comet
[GRID]> crsctl eval activate policy Weekend
Stage Group 1:
-------------------------------------------------------------------------------
Stage Number Required Action
-------------------------------------------------------------------------------
1 Y Server 'comet' will be moved from pools
[ora.frontoffice] to pools [ora.backup]
Y Server 'dancer' will be moved from pools
[ora.frontoffice] to pools [ora.backup]
Y Resource 'ora.raccdb1.crmsvc.svc' (1/1) will be
in state [OFFLINE]
Y Resource 'ora.raccdb1.crmsvc.svc' (2/1) will be
in state [OFFLINE]
2 Y Resource 'ora.raccdb1.backup.svc' (1/1) will be
in state [ONLINE|INTERMEDIATE] on server[comet]
Y Resource 'ora.raccdb1.backup.svc' (2/1) will be
in state [ONLINE|INTERMEDIATE] on server
[dancer]
vixen srvpool: frontofficecomet srvpool: frontoffice
raccdb1
raccdb1_3
raccdb1_4 raccdb1_2
raccdb1_1

What-If with Oracle RAC 12c
What-If evaluation of policy changes – in various levels
[GRID]> crsctl eval activate policy Weekend -admin -l 'serverpools'
NAME = Free
ACTIVE_SERVERS =
NAME = Generic
ACTIVE_SERVERS =
NAME = ora.backoffice
ACTIVE_SERVERS = vixen
NAME = ora.backup
ACTIVE_SERVERS = comet dancer
NAME = ora.frontoffice
ACTIVE_SERVERS = dasher
[GRID]> crsctl eval activate policy Weekend -admin -l 'resources'
--------------------------------------------------------------------------------
Name Target State Server Effect
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.mgmtdb
1 ONLINE ONLINE dasher
ora.raccdb1.backup.svc
1 ONLINE ONLINE comet Started
2 ONLINE ONLINE dancer Started
ora.raccdb1.crmsvc.svc
1 ONLINE OFFLINE Stopped
2 ONLINE OFFLINE Stopped
ora.raccdb1.db
1 ONLINE ONLINE comet
2 ONLINE ONLINE dancer
3 ONLINE ONLINE vixen
ora.raccdb1.hrsvc.svc
1 ONLINE ONLINE vixen
2 ONLINE OFFLINE
--------------------------------------------------------------------------------

Information on each step on the way
Before After
[GRID]> crsctl modify policyset -attr
"LAST_ACTIVATED_POLICY=NightTime“
CRS-2673: Attempting to stop 'ora.raccdb1.crmsvc.svc' on 'dancer'
CRS-2673: Attempting to stop 'ora.raccdb1.crmsvc.svc' on 'comet'
CRS-2677: Stop of 'ora.raccdb1.crmsvc.svc' on 'dancer' succeeded
CRS-2677: Stop of 'ora.raccdb1.crmsvc.svc' on 'comet' succeeded
CRS-2672: Attempting to start 'ora.raccdb1.backup.svc' on 'dancer'
CRS-2672: Attempting to start 'ora.raccdb1.backup.svc' on 'comet'
CRS-2676: Start of 'ora.raccdb1.backup.svc' on 'dancer' succeeded
CRS-2676: Start of 'ora.raccdb1.backup.svc' on 'comet' succeeded
[GRID]> crsctl modify policyset -attr
"LAST_ACTIVATED_POLICY=DayTime"
CRS-2673: Attempting to stop 'ora.raccdb1.backup.svc' on 'dancer'
CRS-2673: Attempting to stop 'ora.raccdb1.backup.svc' on 'comet'
CRS-2677: Stop of 'ora.raccdb1.backup.svc' on 'dancer' succeeded
CRS-2677: Stop of 'ora.raccdb1.backup.svc' on 'comet' succeeded
CRS-2672: Attempting to start 'ora.raccdb1.crmsvc.svc' on 'dancer'
CRS-2672: Attempting to start 'ora.raccdb1.crmsvc.svc' on 'comet'
CRS-2676: Start of 'ora.raccdb1.crmsvc.svc' on 'dancer' succeeded
CRS-2676: Start of 'ora.raccdb1.crmsvc.svc' on 'comet' succeeded

Oracle RAC 12c Collaborate Best Practices - IOUG 2014 version

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (14)

Similar to Oracle RAC 12c Collaborate Best Practices - IOUG 2014 version

Similar to Oracle RAC 12c Collaborate Best Practices - IOUG 2014 version (20)

More from Markus Michalewicz

More from Markus Michalewicz (20)

Recently uploaded

Recently uploaded (20)

Oracle RAC 12c Collaborate Best Practices - IOUG 2014 version