Roy F Swonger, Senior Director, Database Upgrade & Utilities, Oracle, shares how to upgrade hundreds or thousands of databases.

1. How to Upgrade Hundreds or
Thousands of Databases
In a Reasonable Amount of Time
Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Roy Swonger
Senior Director & Product Manager
Database Upgrade & Utilities
Oracle Corporation
How to Upgrade Hundreds or Thousands of Databases

2. Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
How to Upgrade Hundreds or Thousands of Databases 2

3. Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
$> whoami
3
20 years w/Oracle
Previously with DEC Rdb
11+ years managing
Data Pump
Database Upgrade
SQL*Loader
Transportable Tablespaces
How to Upgrade Hundreds or Thousands of Databases
Roy Swonger

4. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Download Slides from the Database Upgrade Blog
 http://blogs.oracle.com/UPGRADE
How to Upgrade Hundreds or Thousands of Databases 4

5. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Supported Versions
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
Oracle 10.1
(GA: Jan 2004)
Oracle 10.2
(GA: Jul 2005)
Oracle 11.1
(GA: Aug 2007)
Oracle 11.2
(GA: Sep 2009)
Oracle 12.1
(GA: Jun 2013)
Oracle 12.2
(GA: xxx 2016)
today
How to Upgrade Hundreds or Thousands of Databases 5
Premier Support Waived Extended Support Paid Extended Support Sustaining Support
Jan 2009 Jan 2012
Jul 2010 Jul 2013
Aug 2012 Aug 2015
Jan 2015
Jun 2018 Jun 2021
Dec 2020May 2017

6. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
We fully understand that things often are
complicated and business critical
How to Upgrade Hundreds or Thousands of Databases 6

7. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
And things can become really complicated ...
How to Upgrade Hundreds or Thousands of Databases 7
 Oracle 8i to 11.2
 26 TB
 Oracle EBS
 5-6 hours max downtime
Good luck!!!

8. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Or more work ...
 Triple hops
How to Upgrade Hundreds or Thousands of Databases 8
↘ Oracle 9.2.0.1
↘ Oracle 9.2.0.4
↘ Oracle 10.2.0.5
↘ Oracle 12.1.0.2

9. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
But even if it does not look complicated at first sight ...
 Too many different patch levels and ...
How to Upgrade Hundreds or Thousands of Databases 9
0
50
100
150
200
250
300
350
400
10.2.0.3.0 10.2.0.4.0 10.2.0.5.0 11.2.0.1.0 11.2.0.2.0 11.2.0.3.0 11.2.0.4.0
2 21 9 10
113
292
382
Databases
54%
(447 out of 829 databases)
have no bug fixing support anymore

10. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Another example ...
 Let's take a closer look
How to Upgrade Hundreds or Thousands of Databases 10
0%4%
5%
2%
3%
17%
35%
32%
2%
Databases
9.2.0.7.0
10.2.0.3.0
10.2.0.4.0
10.2.0.5.0
11.2.0.1.0
11.2.0.2.0
11.2.0.3.0
11.2.0.4.0
12.1.0.2.0
828

11. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Don't maintain a zoo!
How to Upgrade Hundreds or Thousands of Databases 11
 135 Oracle databases
– 7 different Oracle releases
– 13 different Oracle patch levels
 6 different OS vendors
– 15 different OS versions
2% of all databases
under bug fixing support

12. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Recommendation
 Don't sit it out
– It will just get more complicated and risky
 Beware of application dependencies
 Establish constant database inventory monitoring
How to Upgrade Hundreds or Thousands of Databases 12

13. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Where should you start?
How to Upgrade Hundreds or Thousands of Databases 13

14. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Project Approach?
Planning
Data
Gathering
Testing
Upgrade/
Migration
Go-Live
Quarterly
Patching
How to Upgrade Hundreds or Thousands of Databases 14

15. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
You need a plan
How to Upgrade Hundreds or Thousands of Databases 15
Week T-10 Migration Initiation, & Plan
Baseline & Migration
Migration & Unit test initiation
Unit Test Complete
Application & Performance Test Start
Application & Performance Tests
Application & Performance Tests
Application Testing Complete and Application Freeze
Week T- 9
Week T-8
Week T-7
Week T- 6
Week T-5
Week T-4
Week T-3
Week T-2
Week T-1
Cutover Plan & Tests
Migrate Go/No go
Migration Weekend

16. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
You need to know Which / Where / Who / When
 Database inventory
 OS inventory
 Depending applications, certification
 Responsibilities
 Desired time frame
How to Upgrade Hundreds or Thousands of Databases 16

17. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Build a tracking app – Use APEX
How to Upgrade Hundreds or Thousands of Databases 17

18. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Build Workstreams and Database Groups
How to Upgrade Hundreds or Thousands of Databases 18
Estate
Analysis
Workstream 1
Workstream 2
Workstream 3
Upgrade/Migration Group 1
Upgrade/Migration Group 2
Upgrade/Migration Group 3
Post Migration
Monitoring,
Troubleshooting,
and Tuning Team
Production
Production with
additional monitoring

19. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
General Test Guidelines
Upgrade/Migration Process
Post-Upgrade Functionality
Post-Upgrade Performance
ProdLoad
19How to Upgrade Hundreds or Thousands of Databases

20. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Testing Effort
 Relation between test completeness and costs
Costslow high
Completeness0%100%
How to Upgrade Hundreds or Thousands of Databases 20

21. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Use the Right Tools: AWR Diff Reports
AWR Diff
Report
Migrate
R2
10.2.0.4 12.1.0.2
Snapshot Interval Snapshot Interval
IBM AIX SOLARIS
Compare
21How to Upgrade Hundreds or Thousands of Databases

22. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Use the Right Tools: Database Replay
Database Replay
Migrate
R2
10.2.0.4 12.1.0.2
Capture Workload Replay Workload
Compare – then restore
and run again with changes
IBM AIX SOLARIS
Preprocess Workload
22How to Upgrade Hundreds or Thousands of Databases

23. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Use the Right Tools: SQL Performance Analyzer
SQL Performance Analyzer
STS
Capture SQLs
into a STS
Load STS
STS
Evaluate, compare and tune –
over and over again!
Migrate
R2
10.2.0.4 12.1.0.2
IBM AIX SOLARIS
23How to Upgrade Hundreds or Thousands of Databases

24. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Use the Right Tools: SPA Resulting Reports
 Example: Report showing regressed statements and plans
24How to Upgrade Hundreds or Thousands of Databases

25. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Testing Tools – Hand-in-Hand
How to Upgrade Hundreds or Thousands of Databases 25
Collect execution plans
before upgrade
Verify them with SQL
Performance Analyzer
Regressed plans?
SQL Plan Management
SQL Tuning Advisor
Verify functionality and
performance with
Database Replay

26. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Now RELAX ... Stay Calm ...
And open an SR with Oracle Support in case of real trouble
How to Upgrade Hundreds or Thousands of Databases 26
Picture taken from: https://aycw.wordpress.com/2015/06/10/young-peoples-attitudes-toward-work/

27. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
And don't forget:
The next patching day is right in front of you!
Oracle PSUs get delivered on a quarterly basis:
20-OCT-2015 / 19-Jan-2016 / 19-Apr-2016 / 19-Jul-2016
How to Upgrade Hundreds or Thousands of Databases 27

28. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Use Oracle EM Lifecycle Management Pack
How to Upgrade Hundreds or Thousands of Databases 28

29. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
With a little help from ... Real Application Testing
Upgrade over 300 databases at CSX
How to Upgrade Hundreds or Thousands of Databases 29

30. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Real World Checkpoint
 CSX
– One of the nation‘s leading transportation suppliers
 Encompasses 21,000 miles of tracks
in 23 states, from Florida to Ontario
 HQ in Jacksonville, Florida
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks
How to Upgrade Hundreds or Thousands of Databases 30

31. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
 Upgrade and migration of 304 databases to 11.2.0.2 over a
24 month period (2011-2012)
– 1/3 production, 2/3 test and development systems
– Source versions: Oracle 10.2.0.4 and a few Oracle 8i/9i
– Single Instance on IBM AIX  RAC on Linux
 Plan to use Real Application Testing
– Guarantee quality and mitigate the risk of upgrading
 Define standards
– Install 11.2.0.2 and patches and apply latest PSU
Real World Checkpoint
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks
How to Upgrade Hundreds or Thousands of Databases 31

32. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
 Limited downtime on some business critical systems
 Testing cycles vary from 1-6 months depending on
complexity and criticality
 Tight on available testing resources
– Time
– People
– Systems
 New to Real Application Testing
Real World Checkpoint
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks
How to Upgrade Hundreds or Thousands of Databases 32

33. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
 Close alliance with Oracle's Upgrade Development Group and
Oracle's RAT Pack
– CSX agreed to become an official
Oracle Database 11.2 Reference Customer
 Monthly status meetings
 Onsite training
 Live upgrade and RAT demo
– Cloning databases methods for use with RAT
Real World Checkpoint
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks
How to Upgrade Hundreds or Thousands of Databases 33

34. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
 Upgrade and migration methods used:
 Policy:
– Install Oracle SW with latest patches and PSU
– Combine HW refresh and OS upgrades
 Internal application to track owner, status, success
80%
20% Data Pump for OS Migration
DBUA for Upgrades
Real World Checkpoint
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks
How to Upgrade Hundreds or Thousands of Databases 34

35. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
 Yes –project took 2 years
– 88% of all databases upgraded within 18 months
– Some systems took longer due to business and/or resource
constraints
 Performance comparison mechanisms are very helpful
 Regressions found and fixed before upgrade
 RAT helped a lot to predict workload performance and
ensure careful testing
Real World Checkpoint
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks
How to Upgrade Hundreds or Thousands of Databases 35

36. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
 More Information?
https://blogs.oracle.com/UPGRADE/entry/csx_corporation_upgrades_databases_2x
Real World Checkpoint
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks
How to Upgrade Hundreds or Thousands of Databases 37

37. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
A small team and an excellent plan
Upgrade 300 databases at Mobiliar Insurance, Switzerland
How to Upgrade Hundreds or Thousands of Databases 38

38. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Real World Checkpoint
 Swiss Mobiliar
– Switzerland's most personal insurer
– Founded 1826 in Bern, oldest Swiss insurance
– Legal form:
 Cooperative association (mutual company)
– Over 1.7 million persons and firms insured
– Switzerland's no.1 insurer for:
 Businesses, term life insurances, households
– 80 general agencies at 160 locations
– Over 4,400 employees and 325 trainees
– Awarded Most Trusted Insurance Brand in
Switzerland for 13 consecutive years
39
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks
How to Upgrade Hundreds or Thousands of Databases

39. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Real World Checkpoint
 Upgrade 289 databases
– Oracle Database 11.2.0.3 to Oracle Database 12.1.0.2
– 80 production databases
 Oracle Cloud Control
 AWR Warehouse (home-made)
 Shell scripts
 Oracle Restart with ASM
 Oracle In Memory
 OID
40
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks
How to Upgrade Hundreds or Thousands of Databases

40. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Real World Checkpoint
 Project timeline: 24 months
– Goal: 18 months
41
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks
How to Upgrade Hundreds or Thousands of Databases
0
5
10
15
20
25
30
35
40
45
Jan-15 Feb-15 Mar-15 Apr-15 May-15 Jun-15 Jul-15 Aug-15 Sep-15 Oct-15 Nov-15 Dec-15
Migrations per month
Migrationen

41. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Real World Checkpoint
 Small DBA Team
– Only 3.8 FTE + 1 newbie
 2 major software releases of Mobiliar Applications per year
resulting in a full month of code freeze
 Bug fixing support for Oracle 11.2.0.3 ended Aug 27, 2015
 Highly complex applications
– Queries with more than 1000 bind variables and 250 outer joins
 One core team member broke his leg in
September 2015 and did not return to work yet
42
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks
How to Upgrade Hundreds or Thousands of Databases

42. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Real World Checkpoint
 All queries, execution time and execution paths are tracked on a self
made AWR Performance DWH
 Possibility to compare performance before and after the upgrade on
query level
 Performance tests performed by application owner
 Regression tests done during the testing phase of the Mobi Software
Release
– PSU RDBMS 12.1.0.2.4, PSU GRID 12.1.0.2.2
– Two Oracle Homes on the same server
– Upgrade with catctl.pl embedded into home-built shell script
43
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks
How to Upgrade Hundreds or Thousands of Databases

43. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Real World Checkpoint
 Project Monitoring with Excel Sheet
44
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks
How to Upgrade Hundreds or Thousands of Databases
0
50
100
150
200
250
300
350
Jan-15 Feb-15 Mar-15 Apr-15 May-15 Jun-15 Jul-15 Aug-15 Sep-15 Oct-15 Nov-15 Dec-15
Jan-15 Feb-15 Mar-15 Apr-15 May-15 Jun-15 Jul-15 Aug-15 Sep-15 Oct-15 Nov-15 Dec-15
DBs 12c 0 7 10 18 26 62 105 126 136 152 157 170
DBs 11g 289 282 279 271 263 227 184 163 153 137 132 119
Oracle Database Swiss Mobiliar
DBs 11g DBs 12c

44. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Real World Checkpoint
 Automated upgrade via home-built scripts:
45
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks
How to Upgrade Hundreds or Thousands of Databases

45. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Real World Checkpoint
 Live? And alive?
– Yes! First system went live on Feb 4, 2015
– Current status (Oct 12, 2015):
 152 out of 289 databases live on Oracle Database 12.1.0.2
 53% upgraded already
 Project perfectly on schedule
46
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks
How to Upgrade Hundreds or Thousands of Databases
137
152
No. Databases
Oracle. 11.2.0.3 Oracle 12.1.0.2

46. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Real World Checkpoint
 No major issues after go live
– Issues caught during testing, e.g.:
– Bug 20776435: PARSE OF INMEMORY QUERY IS SPINNING ON CPU
 Query with 250 Left Joins
 Workaround:
alter session set "_optimizer_inmemory_table_expansion"=false;
– Bug 19677469: NO QUERY REWRITE
 Workaround:
CREATE MATERIALIZED VIEW …. SELECT /*+ MV_MERGE */ …
or:
alter system set "_fix_control"='10145667:OFF';
47
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks
How to Upgrade Hundreds or Thousands of Databases

47. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Real World Checkpoint
 Oracle In-Memory boosts performance
– Read on: http://tinyurl.com/Mobi12c
48
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks
How to Upgrade Hundreds or Thousands of Databases

48. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Real World Checkpoint
"The entire upgrade project of our 300 Oracle databases at
Die Mobiliar is running very well.
When we catch issues, we fix them before going live.
And features such as Oracle In-Memory add a lot of
performance boost to some of our applications.
Especially the Oracle Customer Reference Program was a
great help and added a lot of value to our project's progress.
We are very happy and satisfied with the upgrade process
and the reliability and performance of Oracle Database 12c".
Paolo Kreth, Group Manager Databases, Die Mobiliar
49
Customer
Project
Constraints
Preparation
Upgrade
Success?
Remarks
How to Upgrade Hundreds or Thousands of Databases

49. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Famous Last Words
Conclusion
How to Upgrade Hundreds or Thousands of Databases 50

50. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Conclusion
 If you have a complicated environment, you are not alone!
 But, you can do it:
– Plan your effort
– Use the right tools
– Track and
report progress
How to Upgrade Hundreds or Thousands of Databases 51

51. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. |
Download Slides from the Database Upgrade Blog
 http://blogs.oracle.com/UPGRADE
How to Upgrade Hundreds or Thousands of Databases 52

52. Copyright © 2015 Oracle and/or its affiliates. All rights reserved. | How to Upgrade Hundreds or Thousands of Databases 53
Classroom Training
Learning Subscription
Live Virtual Class
Training On Demand
Keep Learning with Oracle University
education.oracle.com
Cloud
Technology
Applications
Industries

53. Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | How to Upgrade Hundreds or Thousands of Databases 54