1. 10.2.0.5 Patch Set Apply on
Oracle Clusterware,
ASM and RDBMS on
2-Node RAC Linux x86_64
Version: V0.1
By
Raheel H Syed
10g OCP Certified
Blog: http://raheeldba.wordpress.com
Twitter: https://twitter.com/raheelhsyed
LinkedIn: http://in.linkedin.com/pub/raheel-hashmi-syed/33/34a/226

2. 10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
Pre-installation Tasks
RACNODE1 and RACNODE2 have separate homes for ASM and Oracle Home and
CRS Home. All homes will be patched on both the nodes
Existing CRS_HOME, ASM_HOME & ORACLE_HOME version is 10.2.0.4
+ASM1:/app/asm
PROD1:/app/oracle/10.2.0/db_1
CRS home  /app/crs/home_1
1. Backup CRS, ASM, DB Home and inventory for both the nodes
- Use tar to backup the homes as ‘ tar –cvf ’
2. Check opatch lsinventory on CRS, ASM and Oracle Home. Save the output.
3. Check cluster health, crs activeversion and software version on both the nodes
and save the output.
$ crsctl check crs
$ crsctl query css activeversion
CRS active version on the cluster is [10.2.0.4.0]
$ crsctl query css software version
CRS software version on node [racnode1] is [10.2.0.4.0]
4. Upgrading Installations that use Java Virtual Machine or Oracle interMedia
- If the database uses JVM or Oracle interMedia, then install the oracle database
10g Products installation type from 10g Companion CD before installing
10.2.0.5 Patchset. MOS ID: 293658.1
5. Update Oracle Time Zone Definitions
Raheel H Syed 2

3. 10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
(A) When patching from a lower 10.2 release or upgrading from 10.1.0.x directly
to 10.2.0.5, check the current version of the Oracle time zone definitions with the
following command:
SELECT version FROM v$timezone_file;
>>> If this query reports version 4, no action is required.
>>> If this reports a version lower or higher than 4, see My Oracle Support
document 1086400.1 Actions for the DSTv4 update in the Release 10.2.0.5
patchset.
(No Action required in our case as version is 4)
6. chmod -R +w CRS_HOME/inventory/Templates/*
7. Save all db links - Incase downgrade is required you can drop the db links and
recreate after if downgrade is required
spool dblinks_pre_upgrade.lst
SELECT
'create '||DECODE(U.NAME,'PUBLIC','public ')||'database link
'||CHR(10)
||DECODE(U.NAME,'PUBLIC',Null, U.NAME||'.')|| L.NAME||chr(10)
||'connect to ' || L.USERID || ' identified by '''
||L.PASSWORD||''' using ''' || L.host || ''''
||chr(10)||';' TEXT
FROM sys.link$ L,
sys.user$ U
WHERE L.OWNER# = U.USER#;
spool off
8. Check invalid objects
spool invalid_pre_upgrade.lst
select substr(owner,1,12) owner,
substr(object_name,1,30) object,
Raheel H Syed 3

4. 10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
substr(object_type,1,30) type, status from
dba_objects where status <> 'VALID';
spool off
Run utlrp.sql and re-query invalid objects
The last queries (invalid objects check) will lists all objects that cannot be compiled
before upgrade in file 'invalid_pre.lst'
9. Create pfile from spfile AND backup tnsnames.ora and listener.ora
10. Check AUD$ table, if auditing is enabled then upgrade performance will be
affected
SELECT count (*) FROM SYS.AUD$;
count (*)
---------
0
MOS Id: 979942.1
(No action required in our case as NO auditing is enabled)
11. Verify database components, status and version from dba_registry. Save the
output.
set pages 500
set lines 120
col COMP_NAME for a40
select comp_name, status, version from dba_registry;
12. Backup the database. Using RMAN, backup the full database and archive log in
compressed format to disk. Though we had tape backups configured, we opted
for disk backups as backup completion time to tapes varied to 4-6 hours. Disk
backup completed in 2 hours.
#!/usr/bash
export NLS_DATE_FORMAT="DD-MON-YY HH24:MI:SS"
Raheel H Syed 4

5. 10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
rman target / log /tmp/before_upgrade_full_db.log<<EOF
run
{
allocate channel ch001 device type disk;
allocate channel ch002 device type disk;
allocate channel ch003 device type disk;
allocate channel ch004 device type disk;
BACKUP AS COMPRESSED BACKUPSET DATAFILE 1,2,3,4,5,22,23,24
format '/app/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';
BACKUP AS COMPRESSED BACKUPSET DATAFILE
25,26,28,29,31,32,33,34 format '/app/prod1/before_upgrade/
%d_%s_%p_%c_%t.rmn';
BACKUP AS COMPRESSED BACKUPSET DATAFILE
35,36,37,39,40,42,43,44 format '/app/prod1/before_upgrade/
%d_%s_%p_%c_%t.rmn';
BACKUP AS COMPRESSED BACKUPSET DATAFILE
45,46,47,48,49,50,51,52 format '/app/prod1/before_upgrade/
%d_%s_%p_%c_%t.rmn';
BACKUP AS COMPRESSED BACKUPSET DATAFILE
53,54,55,56,58,59,60,61 format '/app/prod1/before_upgrade/
%d_%s_%p_%c_%t.rmn';
BACKUP AS COMPRESSED BACKUPSET DATAFILE
62,63,64,65,66,67,68,69 format '/app/prod1/before_upgrade/
%d_%s_%p_%c_%t.rmn';
BACKUP AS COMPRESSED BACKUPSET DATAFILE
70,71,72,73,74,75,76,77 format '/app/prod1/before_upgrade/
%d_%s_%p_%c_%t.rmn';
BACKUP AS COMPRESSED BACKUPSET DATAFILE
78,79,80,81,82,83,84,85 format '/app/prod1/before_upgrade/
%d_%s_%p_%c_%t.rmn';
BACKUP AS COMPRESSED BACKUPSET DATAFILE 6,7,8,9,10,11,12,13
format '/app1/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';
BACKUP AS COMPRESSED BACKUPSET DATAFILE
14,15,16,17,18,19,20,21 format '/app1/prod1/before_upgrade/
%d_%s_%p_%c_%t.rmn';
BACKUP AS COMPRESSED BACKUPSET DATAFILE 27,30,38 format
'/app1/prod1/before_upgrade/%d_%s_%p_%c_%t.rmn';
backup as backupset format '/app1/prod1/before_upgrade/%d_
%s_%p_%c_%t.ctl' current controlfile
tag='Before_upgrade_CTL';
sql 'alter system archive log current';
sql 'alter system archive log current';
Raheel H Syed 5

6. 10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
sql 'alter system archive log current';
BACKUP AS COMPRESSED BACKUPSET format
'/app1/prod1/before_upgrade/%d_%s_%p_%c_%t.arc' archivelog
all tag='Before_upgrade_ARCH';
backup as backupset format '/app1/prod1/before_upgrade/
%d_before_upgrade_AFTER_ARCH_%s_%p_%c_%t.ctl' current
controlfile tag='Before_upgrade_CTL';
release channel ch001;
release channel ch002;
release channel ch003;
release channel ch004;
}
quit
EOF
Note: I have explicitly specified datafiles for each compressed backupset, this is
because there was not enough space at /app mount point.
Database size 1230 GB and compressed backup size on disk 139 GB.
Upgrade Clusterware
1. Resolve any outstanding unresolved distributed transactions:
SQL> select * from dba_2pc_pending;
If this returns rows then run the following
SQL> select local_tran_id from dba_2pc_pending;
SQL> execute dbms_transaction.purge_lost_db_entry ('<input_id>');
SQL> commit;
2. Disable the jobs that run from scheduler before upgrading the database.
 Downtime starts here.
Note: In our case all the production jobs were scheduled from
DBMS_SCHEDULER hence we have disabled the jobs.
3. Stop database, ASM and nodeapps on both the nodes.
$ srvctl stop database –d <>
$ srvctl stop asm –n node1
$ srvctl stop asm –n node2
Raheel H Syed 6

7. 10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
$ srvctl stop nodeapps –n node1
$ srvctl stop nodeapps –n node2
4. Download and unzip patch p8202632_10205_Linux-x86-64.zip
5. We have used VNC viewer to install this patch set, most of the Linux system
comes with vnc service running on the system. Please check with your SA if it’s
not running. To check if vnc service is running on the system or not, issue the
following command
$ ps –ef | grep vnc
If the output shows …… oracle ….. <your hostname>:<number> then it means
you have vnc session running under oracle user, eg: racnode1:1 or racnode1:2
If VNC service is not running then you can do this, it worked for me
As ‘Oracle’ user:
$ cd /usr/bin
$ ls –lrt vnc*
$ ./vncserver
It will ask for password, provide the password and verify.
6. CD to Disk1 and Invoke runInstaller and select CRS HOME from the path.
Follow the instructions as displayed on the screen.
OR
Execute the following to ignore pre-requisite checks of parameters (If required)
$ ./runinstaller –ignoreSysPrereqs
(A)
Raheel H Syed 7

8. 10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
(B) Select CRS_HOME
(C) Click next by default all the nodes in cluster are selected as below
Raheel H Syed 8

9. 10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
(D) Warning: Backup /etc/sysctl.conf and modify fs.file_max=101365 (see below)
(E)
Raheel H Syed 9

10. 10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
(F) stop crs as ROOT user; As ROOT user run CRS_HOME/install/root102.sh
Raheel H Syed 10

11. 10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
7. Issues faced during upgrade:-
- Prerequisite check failed as fs.file-max = 655367, required value 101365.
Asked SA Admin to backup the /etc/sysctl.conf and modify the parameter.
- Error in writing to CRS_Home:/app/oracle/product/10.2.0/crs/OPatch
Change permission as $ chmod +w opatch
8. The script will automatically start crs daemon on the patched node upon
completion
9. crs_stat -t (or) crs_status -t on completed node
10. Proceed to node2 and repeat.
This completes CRS upgrade to 10.2.0.5
11. Check crs softwareversion and activeversion on both the nodes. On successful
upgrade crs softwareversion and activeversion should be at 10.2.0.5
$ crsctl check crs softwareversion
$ crsctl check crs activeversion
Upgrade ASM and Oracle Home
1. Stop database, ASM and nodeapps on both the nodes.
$ srvctl stop database –d <>
$ srvctl stop asm –n node1
$ srvctl stop asm –n node2
$ srvctl stop nodeapps –n node1
$ srvctl stop nodeapps –n node2
2. Invoke runInstaller and select ASM Home (ASM is separate home so we will
have to upgrade ASM home first and then ORACLE_HOME.
Raheel H Syed 11

12. 10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
(B) Select ASM home
Raheel H Syed 12

13. 10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
Click Next
Click Next
Raheel H Syed 13

14. 10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
Click OK and
Run the script root.sh on node1 and then on node 2 as root user click OK
Raheel H Syed 14

15. 10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
Click Exit
3. Error during upgrade:-
- When OUI was trying to copy files in
/app/oracle/product/10.2.0/db_1/jdk/bin/java (Text file busy)
I have closed and started the vncviwer, that resolved the issue
- Oracle Configuration Manager Configuration failed with
oracle.sysman.ccr.configCCR.ConfigCCRPlugIn. Skip this step.
4. Execute root.sh as root on first node and then on second node.
5. This completes ASM Home upgrade to 10.2.0.5
6. $ASM_HOME/OPatch/opatch lsinventory. Save the output.
7. Once the ASM Home is upgraded, invoke runInstaller and select ORACLE
HOME to upgrade ORACLE HOME.
8. $ORACLE_HOME/OPatch/opatch lsinventory. Save the output
Raheel H Syed 15

16. 10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
Upgrade Oracle Database
1. Start ASM on both the nodes.
$ srvctl start asm -n node1
$ srvctl start asm -n node2
2. srvctl start instance -d DB -i DB1 -o nomount
3. SQL> ALTER SYSTEM SET CLUSTER_DATABASE=FALSE SCOPE=spfile;
4. Create pfile from spfile.
Edit pfile and comment all obsolete parameters. (Eg: sql_trace)
5. srvctl stop instance -d DB -i DB1
My advice – Run the below steps 6 & 7 in vnc, even in worst case if connectivity
is lost your vnc session is still running in background
6. As ‘Oracle’ User, check or set $ORACLE_HOME, $ORACLE_SID, $PATH
$ sqlplus “/as sysdba”
7. SQL> startup upgrade
SQL> spool upgrade_10205.log
SQL> @?/rdbms/admin/catupgrd.sql
SQL> spool off
SQL> shut immediate
SQL> startup
SQL> @?/rdbms/admin/utlrp.sql
8. SQL> SELECT COMP_NAME, VERSION, STATUS FROM
SYS.DBA_REGISTRY;
9. SQL> shut immediate;
Raheel H Syed 16

17. 10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
10. $ srvctl start instance -d DB -i DB1 -o nomount
11. SQL> ALTER SYSTEM SET CLUSTER_DATABASE=TRUE SCOPE=spfile;
12. $ srvctl stop instance -d DB -i DB1
13. $ srvctl start database -d DB
$ srvctl start nodeapps -n node1
$ srvctl start nodeapps -n node2
14. Health check on crs. $ sh crs_status.sh
Apply PSU patch 13343471 - 10.2.0.5.6 to Oracle Home
1. PRODUCTION nodes RACNODE1 and RACNODE2
2. Have 2 separate homes for ASM and Oracle Home.
3. Apply OPatch patch 6880880 to apply 10.2.0.5.6.
- Backup the existing OPatch directory and then unzip p6880880_102000_Linux-
x86-64_Opatch_4_PSUJan2012.zip in the Oracle Home
- $opatch version
- Output should be like this: 10.2.0.5.1
4. Stop database, asm, listener and nodeapps on all the nodes
$ srvctl stop database –d <>
$ srvctl stop asm –n node1
$ srvctl stop asm –n node2
$ srvctl stop nodeapps –n node1
$ srvctl stop nodeapps –n node2
Before applying patch please read “Patch 13343471 - 10.2.0.5.6 Patch Set
Update” readme.html
Raheel H Syed 17

18. 10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
5. unzip p13343471_10205_Linux-x86-64_PSU_JAN2012.zip
$ cd 13343471
$ opatch apply
6. start database and listener.
7. cd $ORACLE_HOME/rdbms/admin
8. @catbundle.sql psu apply
9. @utlrp.sql
10. set pages 500
select * from dba_registry_history;
Check the following log files in $ORACLE_HOME/cfgtoollogs/catbundle for any
errors:
catbundle_PSU_<database SID>_APPLY_<TIMESTAMP>.log
catbundle_PSU_<database SID>_GENERATE_<TIMESTAMP>.log
11. Quick CRS check with custom scirpt: crs_status.sh
12. Enable all the jobs that were disabled in earlier step.
13. Running changePerm.sh Script on an Oracle Database Server Home
14. $ cd $ORACLE_HOME/install
$ sh changePerm.sh and specify the patched server Oracle home location
References
Raheel H Syed 18

19. 10.2.0.5 PATCH APPLY ON_2-NODE_RAC_STEPS July 10, 2012
 Oracle® Database Patch Set Notes 10g Release 2 (10.2.0.5) Patch Set 4 for
Linux x86-64 Part Number E15229-01 - April 2010
 Complete Checklist for Manual Upgrades to 10gR2 [ID 316889.1]
 10.2.0.5 Patch Set - List of Bug Fixes by Problem Type [ID 1088172.1]
 10.2.0.5 Patch Set - Availability and Known Issues [ID 1087991.1]
 How to Log Good Service Requests for Database Upgrade/Migration Issues [ID
732897.1]
 How to Check Oracle JVM is Installed in the Database [ID 397770.1]
Raheel H Syed 19