This is the whitepaper for my Collaborate 13 presentation with the same title. It describes how Pythian completed a migration project of eBS R12 database top ODA (Oracle Appliance Kit v2.2).

1. COLLABORATE 13
OAUG Forum
Copyright ©2013
by Maris Elsins
Page 1
Running E-Business Suite Database on Oracle Database Appliance
Maris Elsins, Yury Velikanov
The Pythian Group Inc.
Oracle Database Appliance (ODA) is a pre-configured, simple setup, good performance engineered
system running an 11gR2 cluster. It is a great choice for small to medium sized DBs, and if you wish it
can be used for Oracle e-Business Suite too. This paper will show you how the standard configuration
of ODA can be adjusted to comply with the specific requirements of e-Business Suite without
sacrificing ODA’s flexibility and supportability. The paper will also share the authors’ experience
migrating, running and maintaining R12.1.3 Oracle e-Business Suite (eBS) database tier on ODA.
Pythian completed a database consolidation project last year. Four databases of versions 10.2.0.3-
10.2.0.5 were upgraded to 11.2.0.3 and migrated to a single ODA. The project was especially
interesting because one of the databases was an e-Business Suite R12 database, and there was no
information on My Oracle Support on how to cope with the specific e-Business Suite requirements
and how to preserve the supportability and ease of maintenance of ODA's technology stack. This
paper is a case study of how Pythian configured the eBS database on ODA v2.2.
Why choose ODA?
ODA is the smallest of Oracle's engineered systems designed for running Oracle Database, but it still
brings a huge set of benefits – simple deployment as it's pre-configured, reliable as it's pre-tested,
highly available as all the components are redundant, and affordable with its 50,000$ (60,000$ for
ODA X3-2) list price and pay-as-you-grow licensing model.
ODA contains two database-server nodes, each configured with two Intel Xeon X5675 6-core
processors, giving total of 24 cores per ODA. Each node has 96 GB of memory and 500Gb of
mirrored storage for the operating system and software installations. The shared storage consists of
twenty 600Gb SAS drives that can be configured using high or normal redundancy, resulting in 4Tb or
6Tb of space for storing the data and backups, and an additional four 73Gb SSD drives triple-mirrored
for online redo logs. The hardware specification of the ODA makes it attractive to many medium-sized
databases, including ones supporting e-Business Suite.
A new Oracle Database Appliance X3-2 version was announced in March 2013. It provides the same
benefits, but the hardware is different. Each database-server node is configured with two Intel Xeon
E5-2690 8-core processors, so the total number of CPU cores is 32. Each node has 256 GB of
memory and 600 GB of storage for operational system and software installations. The shared storage
is built of twenty 900 GB 2.5” SAS-2 drives that provide 6Tb space if triple mirroring is configured and
9Tb with double mirroring. There are four 200 GB SSD drives included for 260 Gb of space for redo
logs configured in ASM high redundancy disk group configuration.
Challenges
Any engineered system will have limitations, and ODA is no exception. These limitations can be
divided into two categories - limitations by design and supportability limitations.
Limitations by design are enforced by the provided hardware configuration. Depending on which
ODA you choose (ODA or ODA X3-2), The computing capacity is limited to a single ODA and the
configuration is non-expandable - it’s not possible to interconnect two ODAs to create a larger cluster.
On the otherhand, even the old version of ODA, with its 96Gb of RAM and 24 CPU cores is more than
enough for most small and medium eBS installations. The limited disk space of 4Tb (6Tb from v2.4)
can partially be addressed by storing backups on NFS-mounted storage over a 10Gbps interface and
in case of X3-2 you have an additional option of adding a Storage Expansion to double the available
capacity.
Supportability limitations - There are strict rules on how to manage the ODA. Oracle only supports
patching and upgrading by using the oakcli (OracleApplianceKitClient) utility. The patches are
designed for a consistent and uniformly patched system, so anything that is installed additionally

2. COLLABORATE 13
OAUG Forum
Copyright ©2013
by Maris Elsins
Page 2
using opatch can cause issues with patching in the future. There are certain situations when Oracle
Support could allow installation of a one-off patch, but it's also noted that these one-offs could be
required to be uninstalled before patching ODA in the future. The Oracle Global Inventory should be
kept clean and unmodified. Additional products should not be installed on ODA, as oakcli relies on the
information stored in the inventory and any unexpected entries can cause issues with patching.
The first publicly available versions of ODA software were quite limited, Oracle releases new software
versions every 3 to 6 months and they add new features. For example: v2.1 supported only a single
11.2.0.2 oracle home, v2.2 gave us Oracle Database version 11.2.0.3, v2.3 introduced support for
multiple oracle homes and allowed running multiple databases of different versions, v2.4 gave us a
choice of normal redundancy for +DATA and +RECO diskgroups (the only option before that was high
redundancy) and, finally, v2.5 brought Oracle Virtualization support to ODA. An organization needs to
have a roadmap on how to plan and upgrade ODA to keep it on a supportable version.
There are also “supportability” challenges from the E-Business Suite side. The specific
requirements of E-Business Suite R12 are listed in the following My Oracle Support notes:
"Interoperability Notes EBS 12.0 and 12.1 with Database 11gR2 [ID 1058763.1]" and "Using Oracle
11g Release 2 Real Application Clusters with Oracle E-Business Suite Release 12 [ID 823587.1]"
(similar notes exist for 11i - 823586.1 and 881505.1). These documents list a number of requirements
that collide with the supportability limitations of ODA:
• Install Oracle Database 11g Products from the 11g Examples CD
• Apply additional 11.2.0.3 RDBMS patches (4247037, 9858539, 12942119, 12960302,
12985184, 13001379, 13004894, 13258936, 13366268)
• Create a dedicated oracle listener for eBS database
• Customized TNS_ADMIN and ORA_NLS10 settings
• Specific initialization parameters
Although the list is not very long, it’s clear the default Oracle Home configuration on ODA is not
suitable for running the eBS database. The next section of the paper describes how Pythian
customized the configuration of ODA to be able to run the e-Business Suite database and to minimize
the negative effect it could leave on the future maintenance tasks.
Preparation the ODA for eBS database
The E-Business Suite database configuration described in the following sections was implemented on
ODA (v1) and not ODA X3-2. But, as X3-2 introduced changes mostly to the Hardware configuration,
the approach should be valid for the new ODA X3-2 too.
A Dedicated Oracle Home was created for the e-Business Suite database by cloning the RDBMS
Oracle home available on ODA in /u01/app/oracle/product/11.2.0.3/dbhome_1. The new Oracle home
was created with path /u01/cst/{DB_NAME}/product/11.2.0.3 – we created a new directory (cst) under
/u01 to store all custom Oracle homes and we used DB_NAME in the path as we planned to create
multiple homes, and each of them could have its own patch requirements. The default Oracle home
was left intact to be able to run other databases in an unmodified configuration.
Additionally it was decided to create a dedicated Oracle Inventory in /u01/cst/oraInventory for all
customized installations to keep The default global inventory clean and avoid the possible negative
effect on future patching of the ODA. The new Oracle home was registered in the new inventory. As
we had two oracle inventories instead of one, we also introduced a need to maintain /etc/oraInst.loc
file manually by adding the correct inventory location depending on the type of maintenance work
being done – it should point to the default inventory (/u01/app/oraInventory) for normal operations and
it should point to the custom inventory for the maintenance activities affecting the eBS database
Oracle home.
By creating a configuration that was completely independent from the default ODA setup, we could
install the requirements of e-Business Suite database in the freshly created Oracle home without
affecting the seeded configuration. The Examples CD for 11.2.0.3 patch set was downloaded from
My Oracle Support (patch 10404530) and installed into the new Oracle home along with all the
required one-off patches.

3. COLLABORATE 13
OAUG Forum
Copyright ©2013
by Maris Elsins
Page 3
Please note the default grid infrastructure Oracle home was used by all the databases running on
ODA, since eBS didn’t have any specific grid infrastructure patching requirements.
From a certification point of view, ODA is an Oracle Linux 5.6+ server cluster running Oracle grid
infrastructure and database versions 11.2.0.2+. Both OS and database versions are certified for
running Oracle e-Business Suite R12.1. It’s also important to note that despite the fact a new Oracle
home was installed manually (a customization of the Oracle Appliance Kit product) all components –
oracle grid infrastructure, oracle database, e-Business Suite, and even the oracle home we cloned –
are still set up in a fully certified and supported configuration.
An important preparation step for going live with ODA is implementation of a monitoring
framework for timely notification about issues. The best way for monitoring the ODA hardware is
by configuring the Auto Service Request (ASR) support feature that is included in the ODA software
bundle. ASR automatically creates Service Requests, provides diagnostic information to Oracle
Support, and notifies administrators about hardware faults. ODA also supports installation of 3
rd
party
agents to manage, monitor, backup, replicate, authenticate, or otherwise act on the database, the
server, or the environment (note ID 1415773.1), giving the opportunity to handle the ODA the same
way as any other server in your IT infrastructure.
It is worth mentioning that one day before our go live activities, one of the HDDs failed, but as there
was triple-redundancy it had a little impact on the project. After a short rebalancing period, ODA
recovered the protection level with one less HDD. A few weeks later a new HDD was added
seamlessly to the running production system.
Database migration and configuration
As part of the migration project to ODA we also had to upgrade the database from 10.2.0.5 to
11.2.0.3. Since the old HW was not very powerful we wanted to utilize the performance of the ODA
to speed up the upgrade process. This requirement revealed another problem – how to migrate
10gR2 database to 11gR2 on ODA? The following options were considered:
• Upgrade the DB on the old hardware and then move to ODA – the option was rejected as it
required too much downtime.
• Install 10gR2 Oracle home and create a physical standby on ODA, then switchover and do
the upgrade on ODA – this approach would allow us to meet the downtime requirements but
the problem was the ASM diskgroups on ODA were shipped with database compatibility
setting of 11.2.0.2, and therefore the 10gR2 RDBMS software couldn’t use them. A
workaround of recreating the ASM diskgroups with compatibility setting of 10.2 was
considered, but was found too intrusive and too risky as it may break ODA supportability.
The chosen upgrade method was based on the fact that RMAN is able to restore backups taken on
an older version of the database into a newer release (MOS ID 369644.1). The backup of the 10.2.0.5
database was restored using 11.2.0.3 software into ASM disk groups with database compatibility
setting of 11.2.0.2. After completing the restore, the archived log apply process was scripted to keep
the target database synchronized with the source. Remaining redo was applied and “alter database
open resetlogs upgrade” was executed to put the database into the upgrade mode at the switchover,
after which we could continue the normal upgrade process on the ODA. The pre-upgrade procedures
still had to be performed on the source (10gR2) database.
Oracle e-Business Suite has requirements for specific initialization parameters listed in My Oracle
Support note “Database Initialization Parameters for Oracle E-Business Suite Release 12 [ID
396009.1]”, but in addition to these it was found that a vanilla database created by “oakcli create
database” command had the following non-default initialization parameters:
• _disable_interface_checking=TRUE
• _ENABLE_NUMA_SUPPORT=FALSE
• _FILE_SIZE_INCREASE_INCREMENT=2143289344
• _gc_policy_time=0
• _gc_undo_affinity=FALSE
• _KGL_CLUSTER_LOCK_READ_MOSTLY=TRUE
• _kill_diagnostics_timeout=140
• _lm_rcvr_hang_allow_time=140

4. COLLABORATE 13
OAUG Forum
Copyright ©2013
by Maris Elsins
Page 4
• db_block_checking='FULL'
• db_block_checksum='FULL'
• db_lost_write_protect='TYPICAL'
• filesystemio_options='setall'
• parallel_adaptive_multi_user=FALSE
• parallel_execution_message_size=16384
• parallel_min_servers=0
• parallel_threads_per_cpu=2
• use_large_pages='ONLY'
• compatible=11.2.0.3
As the parameters had obviously been fine tuned for ODA we set them in our eBS database too.
Clusterware resources for the eBS database and the listener were created to ease the operational
tasks of the database. A dedicated listener was created for the e-Business Suite database using
port pool 1; it’s important to choose other port pool then 0 to avoid port conflicts with the default
listener running on port 1521. The following commands were used to create the listener:
• srvctl add listener -l LISTENER_EBSDB -o $ORACLE_HOME -p 1522
• srvctl setenv listener -l LISTENER_EBSDB -T
TNS_ADMIN=$ORACLE_HOME/network/admin
The important bit for creating the database resource for an eBS database is to set the ORA_NLS10
environment parameter as required in note 1058763.1:
• srvctl add database -d EBSDB -o /u01/cst/EBSDB/product/11.2.0.3 -p
+DATA/EBSDB/PARAMETERFILE/spfile.704.792145311 -a "DATA,RECO,REDO" -n EBSDB
• srvctl setenv database -d EBSDB -t
"TNS_ADMIN=$ORACLE_HOME/network/admin,ORA_NLS10=$ORACLE_HOME/nls/data/9i
data"
• srvctl add instance -d EBSDB -i EBSDB1 -n oda01a-net1
• srvctl add instance -d EBSDB -i EBSDB2 -n oda01a-net1
It’s also critical to note the TNS_ADMIN variable for cluster resources points to the default location in
$ORACLE_HOME/network/admin, but, as AutoConfig creates node specific directories, they have to
be made visible from the default path by adding the following IFILE settings on both database servers:
• echo "IFILE=${ORACLE_HOME}/network/admin/${CONTEXT_NAME}/tnsnames.ora" >
$ORACLE_HOME/network/admin/tnsnames.ora
• echo "IFILE=${ORACLE_HOME}/network/admin/${CONTEXT_NAME}/listener.ora" >
$ORACLE_HOME/network/admin/listener.ora
• echo "IFILE=${ORACLE_HOME}/network/admin/${CONTEXT_NAME}/sqlnet.ora" >
$ORACLE_HOME/network/admin/sqlnet.ora
Impact on Maintenance of ODA and eBS database
The implemented configuration of ODA v2.2 with the eBS database running in a separate oracle
home has been very stable for almost half a year – the pre-configured hardware and Grid
configuration seems to be working. The first installation of an ODA Patch Bundle is still in front of us.
As a dedicated Oracle home and inventory have been created the impact on ODA’s patching
should be minimal. There are, however, a few configuration points that are visible to oakcli utility and
could affect the way it works:
• /etc/oraInst.loc – the inventory_loc setting should point to the default or the custom oracle
inventory depending on the type of the maintenance being performed.
• /etc/oratab – the file lists all the databases running on the server, including the eBS database
running from the customized oracle home. One should avoid running “oakcli update -patch
2.x.x.x.x” to update all components as it upgrades all databases on the ODA by default,
instead oakcli options --infra, --gi and --database should be used to update the software
followed by “--database <db_name>“ for each database running from the default oracle home.
The custom Oracle home will have to be patched manually or it will need to be re-cloned from
the upgraded default oracle home of ODA.

5. COLLABORATE 13
OAUG Forum
Copyright ©2013
by Maris Elsins
Page 5
• Clusterware resources – the cluster resources are globally visible, so if the customized
resources cause trouble they should be temporarily removed.
• Data on ASM diskgroups – it’s unlikely any of ODA Patches would be built in a way that could
harm data on the ASM diskgroups, but precautions should be taken by taking a backup of the
database and storing it outside the ODA.
On the other side, the ODA platform brings some restrictions for maintaining the eBS database as
well:
• Patching – as a completely separate Oracle home is used there are no restrictions to apply
one-off patches, however Patchsets and Patch Set Updates (PSUs) can’t be installed if
the grid Infrastructure is not upgraded because of the dependencies between RDBMS and
grid Oracle homes. The grid home should be upgraded only by applying ODA bundle patches.
• Security Patch Updates – based on My Oracle Support note “Patch Set Updates for Oracle
Products [ID 854428.1]”, the security patches are developed only for the base release, and
once a PSU has been installed, the recommended way to get future security content is to
apply subsequent PSUs. The Oracle Home that was cloned for running e-Business Suite
database was 11.2.0.3.2 (with a PSU applied), therefore the only way to get the security
fixes is to install the latest ODA Patch bundles.
Future Perspective
The future possibilities of running e-Business Suite databases on ODA depend on what features get
included in the coming ODA Patch Bundles. Already we see the trend of transforming the Oracle
Database Appliance into an “Oracle Database and Application Appliance” – the Patch Bundle v2.5
brings virtualization to ODA. Oracle e-Business Suite is already certified with Oracle VM 3.1.1 (see
note ID 465915.1), giving us a great opportunity to place the whole e-Business Suite implementation
on a single ODA box.
For the implementation described in this article (running on ODA bundle patch v2.2), an upgrade to an
ODA software version supporting multiple Oracle homes should be considered. It would allow running
the e-Business Suite database from an ODA-supported Oracle home by installing additional one-off
patches (they would probably need to be uninstalled before ODA patching), giving us a cleaner and
more easily understandable configuration with a single Oracle inventory.
Conclusions
ODA is great HW for running midsized databases, but official support for the e-Business Suite
database tier on ODA is not yet available. Configuring eBS on ODA requires balancing between
supportability of the ODA’s configuration and implementation of all its specific requirements. Our
solution was to create a dedicated Oracle home and inventory for e-Business Suite database to allow
installation of additional one off patches and products from the Examples CD. The impact the chosen
configuration leaves on the further maintenance of ODA and e-Business suite is low and does not
reduce the benefits ODA brings to the operation of e-Business Suite. It is also expected Oracle will
continue to develop the capabilities of ODA and one day perhaps they will provide complete support
for e-Business Suite too.