Automating Google Workspace (GWS) & more with Apps Script
Oracle Database Appliance - RAC in a box Some strings attached
1. Engineered Systems
Oracle database appliance v1 - rac in a box – some strings attached
Fuad Arshad, Walgreens Company
ABSTRACT
The Oracle Database Appliance is an engineered system that is targeted towards small and medium businesses, but also has
all of the bells and whistles required for enterprise deployments. This paper will introduce the Oracle Database Appliance
(ODA), and review the deployment of the ODA from a non sysadmin perspective. This paper includes the deployment and
patching processes and the best practices built into the Oracle Database Appliance. The paper will review the DBA's role in
managing the ODA
TARGET AUDIENCE
This paper is intended mostly for Database Architects and people that deploy Oracle database technologies. This paper will
review various techniques for deploying the Oracle Database Appliance (ODA) successfully as well as how to perform day to
day activities. The various components all tied together provide an effective use case for companies that want to invest in
Engineered Systems, and provide an entry point to understand the management and maintenance considerations for deploying
an engineered system.
EXECUTIVE SUMMARY
Attendees and readers of this paper will learn the following after reading this paper.
Oracle Database Appliance - Overview
Oracle Database Appliance - Predeployment
Oracle Database Appliance - Deployment
Oracle Database Appliance - Patching
Oracle Database Appliance - One Button Database Appliance
INTRODUCTION
The Oracle Database Appliance (ODA) is an entry level engineered system that provides easy access to preconfigured Real
Application Cluster (RAC) database hardware and software. The ODA is accessible to customers at an entry level cost with
software licensing available starting from 2 cores to 24 cores. The ODA is intended to provide companies with an easy to
deploy model that includes all of the hardware and software required to run a complete system into one 4u rack mountable
unit.
INSIDE THE ORACLE DATABASE APPLIANCE
The Oracle Database Appliance (ODA) is a 4u rack mountable unit which consists of two 2u Sun Fire X4370 M2 units
known as Server units (SN). Oracle has provided MOS Note 1385831.1 which is kept fairly up-to-date with the ODA
components and software versions. Below is a snapshot as of Dec 14, 2012 and this paper only focuses on V1 functionality.
Sun X4370M2 System Server Components
(2 Server Nodes per X4370M2)
CPU 2x 6-core Intel Xeon X5675 3.07GHz
1 108
2. Engineered Systems
Memory 96 GB RAM (12 x DDR3-1333 8GB DIMMs)
2x 10GbE (SFP+) PCIe card
Network 6x 1GbE PCIe card
2x 1GbE (Intel 82571) on-board Integrated redundant cluster interconnect
2x 500GB SATA - for operating system
Internal Storage
1x 4GB USB internal
RAID
2x LSI SAS9211-8i SAS HBA
Controller
20x 600GB - 3.5" SAS 15k RPM HDD (Seagate Cheetah) - for RDBMS DATA (any slot
except the top row of disks)
Shared Storage
4x 73GB - 3.5" SAS2 SSDs - for RDBMS REDO (slot in the top row of 4 disks)
SSD from STEC (ZeusIOPS - multi-level cell (MLC) version with SAS interface)
Operating
Oracle Enterprise Linux 5.5 (on ODA 2.1) 5.8 (on ODA 2.2) x86_64
System
The two ODA Server Nodes are packaged in a 4u chassis that includes 2 server nodes as well as direct attached
disk in the front. Below are front and back images of an ODA. looks like from the front and back
The bottom server is Node 0 and the top is Node 1.These will be referred to as oak1 and oak2 in the factory
default configuration. It is very important to keep this picture handy to work with the data center installation
team to ensure they understand how each port is used to cable the ODA to meet your requirements. Cabling
depends on the data center security zone specs that your infrastructure defines. There might be sensitive and non-
sensitive zones as well as various cabling zones. These requirements need to be considered as part of the
installation process. Attached in the Appendix is a sample spreadsheet that outlines the cabling specs needed to
complete an ODA Installation.
2 108
3. Engineered Systems
Callout Label Ethernet Bond Description
1 Power connectors
Eth 4 and Eth 5 are configured as bond1. Eth 6 and Eth 7 are
Eth 7, 6, 5, 4 bond1, configured as bond 2. These ports are used for custom
2 PCIe 1
(left to right) bond2 configurations or for separate backup, disaster recovery, and
network management.
Two 10-GbE ports. In 10-GbE systems, these are connected to the
3 PCIe 0 Eth 8, Eth 9 xbond0
public network.
4 SerMgt Serial connector to Oracle ILOM and system console.
Net 0,Net Two 1-GbE connectors. In 1-GbE systems, these are connected to
5 Eth2, Eth3 bond0
1 the public network.
6 NetMgt Ethernet connection for Oracle ILOM
USB and
7 Used for connecting to system console.
Video
8 Server Node 0
9 Server Node 1
3 108
4. Engineered Systems
As you can see from above, it, is very important to understand the various components. The complete details are available in
the MOS note referenced above, as well as the ODA Install, Configuration and User’s Guide. This paper will only touch the
basic pieces that are important to create the fundamental understanding about the Oracle Database Appliance. Additional
references are available at the end of this paper.
PREDEPLOYMENT OF THE ORACLE DATABASE APPLIANCE
The Oracle Database Appliance requires a lot of pre-deployment tasks to be completed to prepare for installing the ODA.
These pre-deployment steps are standard tasks for deploying a server in a data-center and we are going to walk thru them
from a Database Administrator perspective. The Oracle Database Appliance Owners guide covers most of these details.
In order to get the environment ready, the site needs to meet the environmental & electrical guidelines that are explained in
the Owner’s manual. The ODA weighs about 96 lbs., and utmost care must be taken while handling the hardware. Four-post
racks are compatible. Two-post racks are not compatible. Horizontal and vertical pitch must conform to ANSI/EIA 310-D-
1992 or IEC 60927 standards. An Oracle Database Appliance requires 4 rack units (4RU) of rack space. A rack mounting kit
is shipped as part of the hardware.
Once the hardware is racked and stacked in the data-center floor, the appliance needs to cabled. If 10g network connections
will be used, SFA Adapters will need to be included in the ODA order for attaching the 10g network cables to the appliance.
There is a minimum requirement of 2 power connections and 4 network connections to the public network in order to access
the machine. There are other networks available for use for various purposes (connecting to a NAS, ILOM or Backup
Network). Once all cabling is confirmed, it is very important to create a pre-deployment checklist to ensure that all
appropriate information about the system is captured.An example template can be seen below.
Needed information Your Information
System Name: i.e. sales TSODA1
Database Deployment Type: RAC, RAC One or RAC
Standalone
Internal or External Backup via NFS device External
Root Passwords: default for nodes is welcome1 Leave the default for now
Domain Name: Example.com.com
DNS Server(s): min 1 Dns.example.com
NTP Server(s): optional but recommended Ntp.example.com
Node 1 Name: i.e. sales1 Tsoda11
Node 1 Public IP Address
Node 1 VIP Name: i.e. sales1-vip Tsoda11-vip
Node 1 Public Virtual IP Address
Node 2 Name: i.e. sales2 Tsoda12
Node 2 Public IP Address
Node 2 VIP Name: i.e. sales2-vip Tsoda12-vip
Node 2 Public Virtual IP Address
SCAN Name: sales-scan Tsoda1-scan
SCAN IP Addresses: min 2
Netmask:
Gateway IP:
ILOM Names: ie. sales1-ilom Tsoda11-ilom
ILOM IP Address:
ILOM Names: i.e. sales2-ilom Tsoda12-ilom
4 108
5. Engineered Systems
ILOM IP Address:
ILOM Passwords: default is changeme Leave the default for now
ILOM Gateway:
ILOM Netmask:
Database Name: Tsodadb
Database class: very small, small, medium, large, very Small
large
(optional) Proxy Server Name: for Auto Service
Request
Metalink Login Information: User ID/Pwd
Since each Oracle Database Appliance comes with the same configuration, it is very easy to provide a configuration
spreadsheet that includes various configuration specs and networking information to record the right information from the
network and data center teams . An example spreadsheet that can be used to streamline the process is attached below.
Microsoft Excel
Worksheet
It is very important to develop a standard process for deploying the ODA. This includes defining roles and responsibilities for
deploying and managing the ODA. It is very important to understand these organizational responsibilities prior to
deployment. A lot of activities that are required to deploy the ODA require system access and organizational structures
typically have role separation. The ODA deployment activity will need the assistance of all teams within an organization from
infrastructure to system administration if the role separation methodology is used. Conversely the Database Administrator
deploying the ODA needs to understand various additional processes in order to successfully deploy the appliance.
In the Scenario, if the Database team is responsible for deploying the Oracle Database Appliance, the following is a brief
example list of activities that will need to be performed.
1. Determine hardware placement zones.
2. Submit a request to get network ip’s
3. Submit DNS and Firewall requests
4. Work with datacenter team to determine rack space.
5. Ensure cables and adapters are available
6. Read thru the ODA manual
7. Provide datacenter team cabling specs
8. Get the ODA connected to a KVM or access via Serial cable to ensure the hosts are powered on and ILOM is
accessible.
9. Walk thru deployment activities.
DEPLOYMENT OF THE ORACLE DATABASE APPLIANCE
In order to understand how to deploy an Oracle Database Appliance (ODA) in a datacenter, one needs to understand the
appliance itself. As illustrated above, The ODA consists of 2 x86 servers that have direct attached SAS as well as SSD
storage. The environment is fully redundant from a hardware perspective. Storage redundancy is provided by Oracle ASM and
can be triple mirrored or double mirrored (version 2.4 and above) for the SAS disk. The SSD is designed and recommended
to be only used for the on-line redo logs and is triple mirrored regardless of ODA version. Oracle ASM also ensures that all
the extents are mapped appropriately among the available disk to ensure that data extents are balanced. Oracle RAC is used to
ensure any hardware failures on one node will not impact the running of the system as a whole.
5 108
6. Engineered Systems
As we look at the appliance we see 1 host node and 1 Service Processor node per hardware entity. The service processor also
includes an Integrated Lights Out Management (ILOM) console which is used to manage each appliance. Deploying the
ODA starts with getting the ILOM ready and accessible to the network.
DEPLOYMENT OF THE ILOM
The ILOM, or the Integrated Lights Out Manager, is the hardware administration and management brains and is also referred
to as the service processor. The ILOM is used for various purposes in the lifecycle of a host including, but not limited to,
performing a bare metal restore of an ODA. An Oracle Database Appliance contains 2 ILOM’s, or one for each of the two
ODA servers. Configuring the ILOM is step 1 in the process of getting an ODA up and running. There are multiple ways to
get into the ILOM. If you have a KVM device you can connect the KVM to the ODA serial management port to do the
ILOM configuration remotely. The configuration steps for setting up the ILOM directly via the serial port are detailed below.
ACCESSING THE ILOM VIA SERIAL PORT
The ODA ILOM can be accessed via a Serial management (SerMgt) port labeled #4 in the picture above. MOS Note ID
1395445.1 explains the process of connecting to the ODA to configure the ILOM port.
The process requires direct physical access to the serial port , which means, this process will need to be run at the place where
the Oracle Database Appliance is installed. Each ODA ships with an RJ45-serial convertor like the one pictured below.
Using an RJ-45 cable, you can connect the cable to the serial port on your laptop or Datacenter terminal cart and get into the
ILOM using putty. In the case that you do not have access to a serial port, Amazon sells a usb to serial converter cable at
http://amzn.com/B00425S1H8 . The cable looks like below and requires prolific drivers installed on your laptop in order to
use the cable. This cable will be used in conjunction to the rj45-serial converter to connect to the ODA ILOM.
Once the connection cable is connected to the laptop and ODA serial port, check the device manager to see which com port
the USB device is connected to and specify the com port in your Putty connection. This doc will only talk about windows but
the MOS note above includes the Linux configuration as well. Once the port is determined, you can open up putty and
connect to the serial connection.
6 108
7. Engineered Systems
Once the connection is made you will see a login prompt. Once the ODA is cabled, in and power cords are attached, the
ILOM is automatically powered on. You will need to use the root account and the default root password to connect to the
ILOM.
.
An initial configuration of the ILOM should have no IP, and this can be validated by issuing the following command:
# show /SP/network
While the OAK deployment can IP the ILOM, it is always faster to configure the ILOM IP’s using the following method.
This will allow for a faster deployment by using the ILOM to deploy the ODA. You will need the following information to be
able to IP the ILOM for both ODA servers.
1. Public IP Address
2. Netmask
3. Gateway Address
The following command will assign a static IP address to the ILOM
# set /SP/network pendingipdiscovery=static pendingipaddress=1.2.2.65 pendingipnetmask=255.255.255.0
pendingipgateway=1.2.2.254 commitpending=true
After the IP is established you can also use this session to power on each of the database nodes remotely.
# start /SYS
7 108
8. Engineered Systems
Once this is executed you should be able to access the ILOM via ssh or an internet browser, as well as connect to the
database node remotely. The following screen shows browser access to an ILOM.
This is the ILOM console and this confirms the successful configuration of the ILOM. Once the ILOM is successfully
configured you are now ready to configure the Oracle Database appliance database nodes. The ILOM provides a lot of
functionality including fault management and remote access. The functions of the ILOM are beyond the scope of this
document but below you can see a picture of some of the ILOM’s capabilities. We will be using the ILOM in the next section
to remote control into the Database Node to configure the ODA.
DEPLOYMENT OF THE DATABASE NODE
The setup of the ILOM is the first step in deploying an ODA and configuring the two database nodes. As part of the
deployment process you will need to create the deployment configuration files or enter the configuration information during
the deployment process. To validate your configuration, it is often wise to use the offline configurator that is available via the
Oracle Technology Network (OTN) website http://bit.ly/VpakT8. The Offline Configurator allows an administrator to
enter, and validate, the complete set of configuration information prior to starting the deployment. The offline configurator
which is meant currently for Bare Metal installations allows for complete network validation prior to deployment.
8 108
9. Engineered Systems
The first step in the deployment is to access each of the database nodes through the ILOM, using SSH as follows:
The ILOM can also be used to access the database node by launching the remote console through a standard web browser.
The host first needs to be powered on either remotely using the ILOM or through the power button on the front panel on
the Oracle Database Appliance boxes.
Once you launch the remote console. you will be taken to the host.
After logging into the host, you will need to IP each of the two database hosts by running the following command using the
root account:
# /opt/oracle/oak/bin/oakcli configure firstnet
This will allow for public access to the appliance and allow for completion of the deployment. Discussion about patching
will be in a separate section but it is important to ensure that the version of OAK (Oracle Application Kit) you are
deploying is not very far behind.
9 108
10. Engineered Systems
#/opt/oracle/oak/bin/oakcli show version
If you feel that the software needs patching, please do so prior to the deployment as there are some factors like disk
configuration that will change based on the version of the software . These include, but are not limited to, disk
redundancy (double vs. triple mirroring) and the introduction of virtualization to the Oracle Database Appliance. The
discussion about redeploying an ODA is a 3 part blog post series that can be found at http://bit.ly/VpcWQH . Once you
have determined and patched your ODA to the level you want to start out with, the deployment process can start. In the
earlier versions of the OAK software, the only option seemed to be using the GUI interface to deploy the software. In
later versions (2.3 and above), the copy command has been introduced which allows for a scripted approach to
deployment. For a safer GUI deployment it is recommended to use the vnc utility in order to avoid any issues.
As root you can execute
# vncserver :1 will allow for vncserver to run on port 5901.
You will be prompted to setup a password. After which you will gain access to the server via vnc. Below is what a vnc
screen will look like on a vnc connect . You can use publically available free vnc viewers like Tight VNC available at
If you have created a deployment configuration file you can copy it to the first node of the host. You can deploy an ODA
by entering the configuration information manually, but it is a cumbersome process. The screens below will show a
typical deployment using a configuration file prebuilt via the offline configurator
For complete control we will select custom and load the file.
10 108
11. Engineered Systems
You will provide generic network info in the next page or the information will be prefilled form a config file that is previously
created.
11 108
12. Engineered Systems
The Public network information including which interface to use is the next set of information. You can either use the 10G
network (xbond0) or the 1G network (bond0) depending on your network configuration. It is very important that you
consult your network team regarding your configuration and ensure cables are plugged into the right ports.
If you have any other networks to configure like a Backup network/NAS network etc . you will get the oppurtunity to
configure it in the next screen.
The next screen will walk you through the database configuration. Typically you should select a very small database to get a
dummy database in place and create the actual databases that are more customized at a later time.
12 108
13. Engineered Systems
The Next screen walks you thru the ASR setup process. It is essential to get an inderstanding ASR (Automatic Service
Request) functionality and the limitations of setting up the ASR utility via the deployment process. If you have direct access to
the internet from the ODA, the ASR deployment should be fairly simple. If you are accessing the internet via a proxy it is
important to note that the ODA ASR registration only supports port 80 for the proxy setup and does not support using
another port using the deployment GUI. Your ASR deployment might fail if you your proxy uses a different port, due to this
at which point you would have to manually setup the ASR.. Later versions of the ODA software (2.5) fixes the proxy port
issue.
The final decision is to the setup the ACFS file system - or as Oracle calls it: CloudFS. ACFS is a shared file system that can
be used for storing a variety of files. Currently ACFS is a restricted license option. Please consult your licensing docs to get
more information.
In conclusion, you will receive a summary of the choices made and you can being the install. The install can take anywhere
from 45 mins to an hour after which you will have a fully functional RAC cluster. This concludes the deployment of the
Oracle Database Appliance
13 108
14. Engineered Systems
PATCHING THE ORACLE DATABASE APPLIANCE
The Oracle Database appliance (ODA) is unique in the sense that Oracle provides an all-inclusive patching strategy. An
ODA patch includes the following components
Infrastructure
1. Integrated Lights Out Management (ILOM)/ BIOS
2. Oracle Appliance Kit (OAK)
3. Hardware Management Pack HMP
4. Automatic Service Request (ASR)
5. Intelligent Platform Management Interface IPMI
6. Operating SSystem
7. Storage Firmware
Grid Infrastructure
The GI patch allows for patching and/or upgrading the Clusterware and ASM software which are now collectively known as
the grid infrastructure.
Database
The integrated patching allows for patching and/or upgrading the database software. Oracle currently supports 11.2.0.2.x as
well as 11.2.0.3.x on the Database appliance.
Oracle =can also introduce new features in each Oracle Appliance Kit patch. Below is the history of some of the features that
were introduced during each patchset.
HISTORY OF OAK/ODA PATCHES
ODA 2.1.0.1/2.1.0.2
1. Base Release
2. 11.2.0.2.4
ODA 2.1.0.3/2.1.0.3.1
1. 11.2.0.2.5
2. Added ASR ILOM configuration
3. ILOM/BIOS and Firmware fixes
4. 2.1.0.3.1 patches a critical disk flaw
ODA 2.2.0
1. Introduced 11.2.0.3.2 and provided the means to upgrade from 11.2.0.2
2. Upgraded the kernel to UEK 5.8
3. Odachk introduced perform ODA health checks
14 108
15. Engineered Systems
ODA 2.3.0
1. Prerequisite of ODA 2.2
2. Multiple Database Homes now supported
3. 11.2.0.2.7 and 11.2.0.3.3 supported in a multi home environment
4. Oakcli command set enhanced to create databases and database homes
ODA 2.4.0
1. Normal redundancy option for new deployment of DATA and RECO(existing deployments require
redeployment)
2. Trace File analyzer
3. Grid Infrastructure and Database patching is rolling upgradable.
ODA 2.5.0
1. Virtualization as an option.
2. External ASR option
3. ASR Proxy port change
PATCHING
Patching the ODA is performed using the Oracle Appliance Kit (OAK) interface. Each version comes with a
comprehensive readme that describes patching behavior. For the purposes of this document we will talk about applying a
standard patch. As just described, an ODA patch will patch the OS as well as database and clusterware components.
Depending on the criticality of the application, as of ODA 2.4 OS patching still requires downtime. However the GI and
Database patching are both rolling upgradable. It depends on your patching strategy and you might want to take a smaller
downtime to patch your OS and follow it up with the rolling patching for GI and Database components.
Since the ODA patch includes a lot of components it is important to understand each component and the impact it will have
on your system. The patch readme as well as MOS Note 888888.1 does a good job of explaining all the changes in the patch.
It is important that the person applying the patch assess the impact and apply accordingly.
Before starting a patching session it is always important to check the version of the patch deployed.
#/opt/oracle/oak/bin/oakcli show version
The version will help determine which patchsets are necessary. If you are on a pre ODA 2.2 patch set, it is important to first
patch up to 2.2 before moving further. This is due to significant OS changes that were made in ODA patch 2.2.
Once the patch determination has been made it is important to understand your network components to understand how the
patching works. As an example, ILOM and BIOS patching will take significantly longer if your ILOM is on a different
network than the database node public network. Typically this is due to network zoning/segmentation.
If you have such a case it is always better to patch the ILOM outside of the ODA patch for speed. To do so you can follow
the procedures outlined below. Please note below we are assuming a system with ODA 2.3 installed patching to 2.4. The first
3 steps can be done ahead of time and require no downtime is required
15 108
16. Engineered Systems
# ./oakcli unpack -package <absolute_path_to_patch_file p14752755_24000_Linux-x86-64.zip> (On
Both Nodes)
# cd /opt/oracle/oak/pkgrepos/thirdpartypkgs/Firmware/Ilom/SUN/541-4116-03/3.0.16.22.a-es-
r75629/Base (On Both Nodes)
# ipmiflash -v write ILOM-3_0_16_22_a_r75629-Sun_Fire_X4370_M2-rom.pkg force script config delaybios
warning=0 (On Both Nodes)
# ipmitool chassis power cycle (Will bring down the node and does require downtime. Needs to be run on
The infrastructure patch a delay)
each node but with will recognize that the ILOM/BIOS has already been patched and will ignore them in the
patching cycle.nodes come up. You can patch the infra and it will be significantly faster
Once the
# /opt/oracle/oak/bin/oakcli update -patch 2.4.0.0.0 –infra
The downtime related tasks end here if the purpose is patching only the server and not upgrade the grid and database
software. The next step is patching the Grid Infrastructure and the patch is rolling upgradable which means a full
downtime is not required.
# cd /opt/oracle/oak/bin
# ./oakcli update -patch 2.4.0.0.0 –gi
This will patch your grid infrastructure components to 11.2.0.3.4.
The final piece in the patching puzzle is the database and that is rolling upgradable as well. As part of the command you will
be asked which Oracle homes to patch.
# cd /opt/oracle/oak/bin
# ./oakcli update -patch 2.4.0.0.0 –database
This concludes the patching exercise on the ODA. As documented patching and maintaining and ODA is much simpler than
managing and patching a similar set of components separately
.
ONE BUTTON ORACLE DATABASE APPLIANCE
Oracle has used the concept of one button for a lot of activities surrounding the Oracle Database Appliance. The Oracle
Appliance Kit (OAK) is the management brains of the Oracle Database Appliance. The Oracle Appliance Kit Daemon
(oakd) maintains the link between hardware and software in the ODA.
ONE BUTTON DEPLOY
We’ve already discussed deployment but this is one of the key features of the Oracle Database Appliance (ODA). One
button deployment allows for rapidly deploying or re-deploying the ODA with one command.
# cd /opt/oracle/oak/bin
# ./oakcli deploy
16 108
17. Engineered Systems
ONE BUTTON VALIDATE
Oracle Database appliance comes with one button validation which should be performed to ensure that the system is
performing optimally prior to going live. It is very important to understand the various options that the validation tool offers
from the documentation.
# cd /opt/oracle/oak/bin
# ./oakcli validate -h
The table of options below tells you about the various validation commands you can run. It is essential to
execute basic validation on a regular basis to ensure the system is stable.
Oracle Database Appliance Validation Tool Options
Option Purpose
Run all system checks, including Disk Calibration. Oracle recommends you use this command to validate
-a system readiness before deployment. Do not run oakcli validate with this option on a loaded system,
because the Disk Calibration system check can cause performance issues.
-c Run a comma-delimited list of checks. Use this command to run a specific check or list of checks.
Run the default system checks. The default system checks are SystemComponents, OSDiskStorage,
-d
SharedStorage, and NetworkComponents
Create an output file. Provide the name of the file after the -f flag. The output for the validation commands
-f is printed to this file. If you do not specify that output is sent to a file, then it is sent to the Page display
(stdout).
-h Print help information.
-v Provide verbose output.
-V Print the validation tool version.
An example of a system problem can be shown here.
RESULT: Powersupply PS0 V_IN=Disabled I_IN=Disabled V_OUT=0.88 Volts I_OUT=2 Amps
IN_POWER=Disabled OUT_POWER=Disabled – as you can see I_IN and V_IN is disabled which mean there is an
issue
RESULT: Powersupply PS1 V_IN=206 Volts I_IN=3.75 Amps V_OUT=12.08 Volts I_OUT=4.40 Amps
IN_POWER=720 Watts OUT_POWER=670 Watts
A system without a power supply issue would display something like this:
RESULT: Powersupply PS0 V_IN=204 Volts I_IN=1.88 Amps V_OUT=12.08 Volts I_OUT=27.20 Amps
IN_POWER=380 Watts OUT_POWER=330 Watts --
RESULT: Powersupply PS1 V_IN=202 Volts I_IN=1.88 Amps V_OUT=12.08 Volts I_OUT=27.20 Amps108
17
IN_POWER=350 Watts OUT_POWER=340 Watts
SUCCESS: Both the powersupply are ok and functioning
18. Engineered Systems
As you can see, even though the system is functioning due to the redundant nature of the system components. If there is an
issue, then running an oakcli validate can identify the source of the system or hardware problem. These types of issues can be
proactively determined by using ASR (Automatic Service Request).
Validate also provides DiskCalibration data to ensure that the disk is functioning as expected and no disk failures.
RESULT: Random read throughput across all 20 SASDisk = 4037 IOPS
RESULT: Random read throughput across all 4 SSDDisk = 16368 IOPS
RESULT: Random read throughput across all 20 SASDisk = 1844 MBPS
RESULT: Random read throughput across all 4 SSDDisk = 941 MBPS
An issue with disk access or calibration can also be seen below. In this example, a disk was manually dropped from the
diskgroup to simulate this test.
ERROR: Error when running orion for HDD_E0_S01_705927132_MBPS : ORION: ORacle IO Numbers -- Version 11.2.0.3.0
Calibration will take approximately 1 minutes.
Using a large value for -cache_size may take longer.
Error completing IO (storax_aiowait)
ORA-27061: waiting for async I/Os failed
Linux-x86_64 Error: 5: Input/output error
Additional information: -1
A Network Check will look at the network components to validate the network speed and Ethernet ports that are currently
active. Oracle Database Appliance has a one 10G interface per node as well several 1G interfaces. In the example only 1G
interfaces are active.
INFO: Doing oak network checks
RESULT: Detected active link for interface eth0 with link speed 1000Mb/s
RESULT: Detected active link for interface eth1 with link speed 1000Mb/s
RESULT: Detected active link for interface eth2 with link speed 1000Mb/s
RESULT: Detected active link for interface eth3 with link speed 1000Mb/s
RESULT: Detected active link for interface eth4 with link speed 1000Mb/s
WARNING: No Link detected for interface eth5
WARNING: No Link detected for interface eth6
WARNING: No Link detected for interface eth7
WARNING: No Link detected for interface eth8
WARNING: No Link detected for interface eth
99
The oakcli validate one button deployment provides a plethora of diagnostic information that provide insight into the
operation of the Oracle Database Appliance. It should be used regularly, but with caution, to ensure the machine is
functioning as intended.
18 108
19. Engineered Systems
ONE BUTTON STORAGE DIAGNOSTICS
Disks can fail, so it is important to ensure that storage diagnostics are key as part of the one button deployment of the ODA.
Storage diagnostics are a key component in the oakcli command set. From the ODA documentation, below are some of the
commands that are available for storage diagnostic and disk location.
Option Purpose
Locates the disk associated with the ASM disk that you
indicate by its ASM disk name, where:
-h outputs the oakcli locate options
oakcli locate[-h] [disk diskname on|off] disk diskname locates the ASM disk whose name you
designate
on turns on the LED of the specified disk
off turns off the LED of the specified disk
oakcli show controller
Displays the configuration for active cores as defined by
oakcli show core_config_key
the registered core key
oakcli show disk Displays disks
oakcli show diskgroup Displays disk groups
oakcli show expander Displays expanders
oakcli show –h Show help command
oakcli show storage Displays storage device status
Provides a detailed output patch versions for software
and firmware on the appliance
It is very important to understand and execute some of the above commands to ensure the stability and understand the state
of the disk.
#oakcli show disk
NAME PATH TYPE STATE STATE_DETAILS
pd_00 /dev/sdam HDD ONLINE Good
pd_01 /dev/sdaw HDD ONLINE Good
pd_02 /dev/sdaa HDD ONLINE Good
pd_03 /dev/sdak HDD ONLINE Good
Based on the output above, if there is a bad disk, you can work with Oracle Field Services or your internal hardware resources
to replace the disk. The oakcli locate disk helps in ensuring the right bad disk is located and replaced. Also, the show storage
option provides a complete picture of the storage which includes the controllers, expanders as well as firmware version of the
disks.
19 108
20. Engineered Systems
ONE BUTTON ODA HEALTHCHECKS
This piece is new since ODA 2.3.x and is not part of the OAK framework. However it is important to note and use as
needed. Oracle Database Appliance Checks, or ODAchk as the tool is known, is the Oracle Database Appliance version of
the popular RACchk and EXAchk tools.
Most of the OAK built in tools look at hardware and software, but the ODAchk provides more in-depth looks the system
from the database perspective and not the appliance perspective. The tool ensures things like RAC, Grid, ASM as well as
common database configuration parameters are set up right. ODAchk should be run on a regular basis and is recommended
before and after any system change as well as any system patching. It is surprising how sometimes on a running system how
many best practices are not being followed.
$/opt/oracle/oak/odachk/odachk – Please run as oracle software owner.
FAIL => Database parameter DB_LOST_WRITE_PROTECT is NOT set to recommended value on tst12 instance
FAIL => Database parameter GLOBAL_NAMES is NOT set to recommended value on tst12 instance
FAIL => One or more errors or warnings for OS Disk Storage checks
WARNING => One or file generated network and bonding interface checks
There is an html more warnings for with detail information as well as best practices to follow.
ONE BUTTON MANAGEMENT DIAGNOSTIC COLLECTION
The OAK framework comes with its own management and diagnostic collection framework including a cleanup script to re-
initialize the Oracle Database Appliance and establish “pay as you go” Oracle licensing core counts.
Oakcli manage diagcollect – Collect and package diagnostic files for Oracle Support
# cd /opt/oracle/oak/bin
# ./oakcli manage diagcollect – this will collect diagnostic data from CRS and various DB and OAK components
to submit to oracle
Oakcli manage cleanrepo – Remove old versions of ODA patch and version files
#cd/opt/oracle/oak/bin
# oakcli manage cleanrepo --ver 2.4.0.0.0
Can not delete the current or latest version repository.
#oakcli manage cleanrepo --ver 2.3.0.0.0 – to cleanup previous version files
# oakcli update –patch 2.4.0.0.0 –clean – in 2.4 has been enhanced to cleanup temporary files after
patching
CleanupDeploy.pl
If there is ever a need to reimage the box or uninstall all software you can use the CleanupDeploy.pl to do so
1. Log in as root.
2. Change directory to /opt/oracle/oak/onecmd.
3. Run the command cleanupDeploy.pl.
For example:
# ./cleanupDeploy.pl
20 108
21. Engineered Systems
Configuring the Oracle licensing core counts
If you are using the pay as go Oracle Licensing scheme you can use oakcli to apply the core cap and install the key you have
generated on MOS (My Oracle Support).
# oakcli apply –h
Usage:oakcli apply core_config_key <key_file>
# oakcli show core_config_key
Optional core_config_key is not applied on this machine yet !
# oakcli show core_config_key
Host's serialnumber = ABC123
Configured Cores = 20
Trace File Analysis
In Oracle Database Appliance 2.4 a new option for TFA was installed. The Trace File Analyzer is a new framework for the
collection and analysis of trace files that help the proactive collection of errors in real-time mode and automatically collects
the right log files as the problem is happening. More information on TFA can be found in MOS note 1513912.1.
ONE BUTTON PATCHING
This paper already discusses patching but in the one button section it is important to mention the capabilities of the one
button infrastructure to be able to patch all software or individual components using the one button patching capabilities built
into the Oracle Appliance Kit.
# cd /opt/oracle/oak/bin
# ./oakcli update -patch version
While the patching section of this paper discusses more detail, the command above is the but in essence of single button
patching, The command above can patch an appliance.
ONE BUTTON DATABASE DEPLOYMENT
Starting with ODA version 2.3 and above, newer releases of the ODA 2.3 and above , the option is now provided to create
multiple database homes as well as create new databases. There are options to upgrade database as well delete databases and
homes as well.
To create a new database home
# oakcli create dbhome -version 11.2.0.3.3 – will create a new dbhome with 11.2.0.3.3
21 108
22. Engineered Systems
To create a new database
# oakcli create database -db testdb -version 11.2.0.3.3 – will create a database and a new database home
#oakcli create database -db testdb –oh OraDb11203_home2 – will create a new database in an existing dbhome.
To delete a database or database home
# oakcli delete database –db olddb
# oakcli delete dbhome –oh dbhome_name
To. upgrade a database from one version to another (11.2.0.2 to 11.2.0.3)
#oakcli upgrade database {-db <db_names> | -from <source_home>} -to <destination_home>.
SUMMARY
In summary, the Oracle Database Appliance packs a lot of bang for the buck. As the basic model in Oracle engineered
Appliance product line, it is the stepping stone to understand and evaluate if an appliance model is a viable option for a
company. An Oracle Database Appliance provides ease of setup, patching as well as diagnostics. As an administrator that has
previously only had a specific focus, there are a lot of things that an administrator has to be aware of, including the network,
hardware and operating system. Oracle has made it easy for people to manage the appliance as one unit by making various
aspects easier to manage and maintain using the Oracle Appliance Kit (OAK) framework. There are various monitoring
aspects built into the box as well as one button activities to ensure that management of the appliance is simple.
REFERENCES
Oracle Database Appliance - 2.X Supported Versions & Known Issues [ID 888888.1]
Information Center: Oracle Database Appliance [ID 1417713.2]
ODA (Oracle Database Appliance): New Deploy Step-by-Step Short Guide (version 2.3) [ID 1486534.1]
ODA (Oracle Database Appliance): oakcli command reference [ID 1417879.1]
ODA (Oracle Database Appliance): How To Configure xbond,bond1,bond2 post deploy [ID 1422563.1]
ODA (Oracle Database Appliance): The Steps to replace failing disks [ID 1496114.1]
Gathering data for the Oracle Database Appliance server [ID 1358795.1]
ODA (Oracle Database Appliance): Test Plan Outline [ID 1474273.1]
Oracle Database Appliance Setup Poster
TFA Collector- The Preferred Tool for Automatic or ADHOC Diagnostic Gathering Across All Cluster Nodes [ID 1513912.1]
22 108