4. Who am I
ZDLRA - in Action4 21.09.2016
Daniele Massimi, Trivadis AG (CH, Bern)
Principal Consultant daniele.massimi@trivadis.com
Seit 2012 bei Trivadis IMS (Infrastructure Service Management)
Oracle DBA seit >16 Jahre (Exadata ~5 Jahre)
Infrastruktur-Architekturen, Operations & Administration, Upgrades and Migration,
High Availability, Backup & Recovery, Enterprise Manager, Engineered Systems
(Exadata, ODA)
Oracle Certifications
Oracle Certified Professional
Oracle Certified RAC Expert
Oracle Exadata Implementations Specialist
5. Agenda
ZDLRA - in Action5 21.09.2016
1. ZDLRA Functionality
2. Installation and Configuration from bottom up
3. Have a look inside
4. From protected Databases to backups
5. Recovery Capabilities
6. Recovery Appliance Views and Utilities
7. Conclusion
6. ZDLRA - in Action6 21.09.2016
ZDLRA Functionality
7. Common problems with Oracle backups
ZDLRA - in Action7 21.09.2016
RPO depends on archive backup frequency (many minutes or hours)
Backup window and nightly batches compete for resources
High network bandwidth usage for full backups
Low-speed backups and restores
Backup and restore success ratio penalized by shared infrastructure
Critical databases require Data Guard for transaction safety
– Additional space, storage, license and computation required
Expiration of backup set on ML: need to crosscheck and delete expired
Lack of backup validation
…
9. Eliminate data loss – Real Time redo transport
ZDLRA - in Action9 21.09.2016
Real Time redo transport
– Data Guard like but asynchronous
– Changes out of Logbuffer transfered
– Validate and writes to a stage area
Redolog switch on database
– RA converts the staged redo data to a
compressed archived redolog backup
– Writes this archlog backup to catalog
– Ready for future restores
Explicit Archivelog backup no more necessary
Reduces RPO to (near) zero
10. Minimal Backup Overhead
10 21.09.2016
Traditional Backup
– Read/Write for Disk/Tape Backup
– Deduplication
– Compression
– Validation
– Deletion
– Merging
All on database host
Degrade performance
Recovery Appliance
– Offloading operations
– Delta Push
– Delta Store
11. Delta Push
ZDLRA - in Action11 21.09.2016
Delta Push is a highly optimized form of source-
side deduplication
– Through RMAN block change tracking
– No reading of unchanged data
– No commited undo
Two operations on the protected Database
– Incremental "forever" backup
– Real time redo transport
One-time full backup as prerequirement
Afterwards Incremental forever backup
Validate incoming Backups against corrupted
physical blocks
Compress the Backups using special block-level
Algorithm
Backuped Database
Changed Data
Less data, less I/O, less network traffic
12. Delta Store – the “brains” of the Recovery Appliance
ZDLRA - in Action12 21.09.2016
Backup
validates the incoming blocks
compresses, indexes and stores
Creates a Virtual Full Database
Backups
10 times less storage usage
High number of virtual full backups
Higher recover window
Delta Store
Day N
Incr
Day 1 Virtual Full
Day N Virtual Full
Day 1
Incr
Day 0
Full
13. Delta Store – fast restore/recover
ZDLRA - in Action13 21.09.2016
Restore
Recreates a physical full backup out
of a virtual one
No need of transfer and apply
incremental backups
Roll forward with restored redologs
Uses Exadata performance and
scalability
Delta Store
Day 0
Full
Day 1
Incr
Day N
Incr
Day ‘N’ Full Backup
14. Delta Pools
ZDLRA - in Action14 21.09.2016
Chunks inside the Delta Store
Backups will be indexed and stored inside the Delta Pools
Set of datafiles blocks from which RA constucts Virtual Full Backups
Each backuped datafile have his own delta Pool
Automated delta pool space management
– Protection Policy will be inherited to database retention policy
– Delete automatically expired or obseleted Backups
– Periodically reorganising for performance improving
– Delta pool optimization when old blocks are deleted and new incremental Backup
arriving
15. Massive Scale out – Oracle Database Backup Service
ZDLRA - in Action15 21.09.2016
10/100/1000 DBs across datacenters
RMAN-integrated subscription-based
backup storage in the public cloud
RMAN compression and encryption
quickly and easily setup with a
secure, low-cost backup
Flexibility in backup/archive concepts
The Recovery Appliance flexibly
supports both tape and cloud archival
option
16. ZDLRA - in Action16 21.09.2016
Installation and Configuration
from bottom up
17. Installation Steps
ZDLRA - in Action17 21.09.2016
Create the ZDLRA configuration using
the latest OEDA
Start the Re-Imaging and Setup from the first
Computing Node (like an Exadata Setup)
[root@zdldbadm01 linux-x64]# ./install.sh -cf ./WorkDir/zdl-zdl.xml –r 1-16
18. Installation Steps
ZDLRA - in Action18 21.09.2016
Recovery Appliance specific Installation Steps
[root@zdldbadm01 install]# ./ra_install.pl --help
ra_install.pl - Recovery Appliance Installer
You must be logged in as root to run this command.
All steps should be run successfully for install.
Usage: ./ra_install.pl --help | --step=STEP_NUMBER {--install|--uninstall} [--
stdout]
Options:
--help: displays this help message
--step=number: Which step to run
--stdout: Print all messages to stdout rather then the log file
--install: Installs the step [Default]
--uninstall: Uninstalls the step
Step Numbers:
Step 1 - Validation and Configuration Prep
Step 2 - OS Setup
Step 3 - Oracle User Setup
Step 4 - DBFS Setup
Step 5 - Tape Backup configuration
Step 6 - ZDLRA DB Backup Setup
Step 7 - Enable ZDLRA Services
19. Configuration Steps
ZDLRA - in Action19 21.09.2016
Deployment of the Agents on the compute Nodes
Recovery Appliance Discovery
Installation of the Backup Module on any Clients
[oracle@odax30 zdlra]$ java -jar ra_install.jar -dbUser ra_orcl02 -dbPass
manager -host zdlingest-scan -port 1521 -serviceName zdlra -walletDir
/u01/app/oracle/admin/orcl02/xdb_wallet -libDir $ORACLE_HOME/lib
Recovery Appliance Install Tool, build 2015-11-03
Recovery Appliance credentials are valid.
Recovery Appliance wallet created in directory
/u01/app/oracle/admin/orcl02/xdb_wallet.
Recovery Appliance initialization file
/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/raorcl02.ora created.
Downloading Recovery Appliance Software Library from file ra_linux64.zip.
Downloaded 27189305 bytes in 5 seconds. Transfer rate was 5437861 bytes/second.
Download complete.
[oracle@odax30 zdlra]$
20. ZDLRA - in Action20 21.09.2016
Have a look inside
(just particularities)
21. On the computing Nodes
ZDLRA - in Action21 21.09.2016
Metadata Database
– RAC Database (also used for Recovery Catalog (VPC))
– Non Container Database
– DBFS Mount-Points for Backup and Replication purpose
– DBFS added as Cluster Resources
– HTTP Service inside the database will be started for communication between the
clients and the ZDLRA (dbms_ra.startup_recovery_appliance)
oracle@zdldbadm01:~/ [+ASM1] df -h | grep dbfs
dbfs-@repdbfs.local:/
957M 120K 956M 1% /dbfs_repdbfs
dbfs-@obdbfs.local:/ 300G 23G 278G 8% /dbfs_obdbfs
SQL> exec dbms_ra.startup_recovery_appliance;
PL/SQL procedure successfully completed.
22. On the computing Nodes
ZDLRA - in Action22 21.09.2016
ASM Configuration
– Two Diskgroups DELTA (for Backups), CATALOG (Metadatas)
– New special sub-Directory CONTAINER for storing Container Files which contains
the Delta Pools and Delta Stores
– Each Container File as a size of 2 TB
ASMCMD [+delta/zdlra] > ls -l
Type Redund Striped Time Sys Name
Y ARCHIVELOG/
Y CONTAINER/
Y DATAFILE/
Y TEMPFILE/
ASMCMD [+delta/zdlra/CONTAINER] > ls -s
Block_Size Blocks Bytes Space Name
512 4294959104 2199019061248 4398059094016 con.365.920823681
512 4294959104 2199019061248 4398059094016 con.366.920823687
512 4294959104 2199019061248 4398059094016 con.367.920823699
...
23. On the computing Nodes
ZDLRA - in Action23 21.09.2016
Backup of Metadata Database
– Done using Export of RASYS Schema every 2 Hours
– Backup Location on DBFS FS: /dbfs_obdbfs/OSB/ra_export
– Scheduling via "anacron"
[root@zdldbadm01 ~]# cat /etc/anacrontab
SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
#RANDOM_DELAY=45
# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22
#period in days delay in minutes job-identifier command
1 65 cron.daily nice run-parts /etc/cron.daily
7 70 cron.weekly nice run-parts /etc/cron.weekly
30 75 cron.monthly nice run-parts /etc/cron.monthly
[root@zdldbadm01 cron.hourly]# ls -l /etc/cron.hourly/
total 4
-rwx------ 1 root root 409 Nov 10 2015 0anacron
lrwxrwxrwx 1 root root 46 Aug 25 16:20 ra_export.pl -> /opt/oracle.RecoveryAppliance/bin/ra_export.pl
24. On the cell Nodes
ZDLRA - in Action24 21.09.2016
Fortunatelly nothing special
Same appearance as in Exadata Storage Cell
During the Installation the "Write Back" Option on the Flashdisks will be enabled
Little difference on the Grid Disks compared with the Exadata, only 10 Grid Disks
instead of 12 for the CATALOG ASM Disk Group
25. User Model ZDLRA
ZDLRA - in Action25 21.09.2016
Component Account Type User Name Description
EM Cloud Control Cloud Control Super User SYSMAN EM Admin User
EM Cloud Control Cloud Control Admin User specified User to manage RA and protected databases
Recovery Appliance RA Metadata Super User SYS DB Admin User, needed for create RA User
Recovery Appliance RA Admin RASYS RA Schema Owner
Recovery Appliance RA User Account User specified
User on RA for managing Metadata on VPC
of protected databases
Protected Database
Protected Database
Backup Admin
User Account with
SYSBACKUP privileges
User for doing backup, restores and recovery
of protected databases
User Accounts:
26. ZDLRA - in Action26 21.09.2016
From protected Databases
to backups
27. Protected Databases
ZDLRA - in Action27 21.09.2016
Once the Environment meets all the requirements we can add the databases to the
ZDLRA
The requirement are:
• Network connectivity
• Backup Module is installed for every Oracle Home
• Definition of Protection Policies are created (Retention Time)
Block Change Tracking not mandatory, but recommended !!!
Two Methods are possible:
• Enterprise Manager
• Manually DBMS_RA PL/SQL Package
28. Prerequisite for Adding Protected Database
(EM and manually)
ZDLRA - in Action28 21.09.2016
Create an VPD User on the Metadata Database
Grant the Create Session privilege
SQL> create user ra_orcl03 identified by <pwd>;
SQL> grant create session to ra_orcl03;
29. Adding Protected Database with EM
ZDLRA - in Action29 21.09.2016
Add Protected Database on Recovery Appliance Page
– Choose the Protection Policy
30. Adding Protected Database with EM
ZDLRA - in Action30 21.09.2016
Configuring Backup Settings
– Register database
– Add RMAN configuration (configure...)
– Enabling Redo Transport
(add parameter to spfile and restart DB)
31. Scheduling Protected Database with EM
ZDLRA - in Action31 21.09.2016
Protected Database is ready for backing up
– We need a Schedule
32. Scheduling Protected Database with EM
ZDLRA - in Action32 21.09.2016
Configure the repeating interval for
the Incremental-Forever Backups
Configure the Archive Log Backups and
deletion rule
33. Scheduling Protected Database with EM
ZDLRA - in Action33 21.09.2016
Also useful as Script Generator here the Script is not modifiable !!!
35. Adding Protected Database manually
ZDLRA - in Action35 21.09.2016
Add Database for the protected database
Grant access to Virtual Private Catalog
Configure the RMAN Channel for ZDLRA
begin
dbms_ra.add_db (
db_unique_name => 'orcl03',
protection_policy_name => 'bronze',
reserved_space => '250G');
end;
begin
dbms_ra.grant_db_access (
db_unique_name => 'orcl03',
username => 'ra_orcl03');
end;
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U' PARMS
"SBT_LIBRARY=/u01/app/oracle/product/12.1.0.2/dbhome_1/lib/libra.so,
ENV=(RA_WALLET='location=file:/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/ra_wallet
credential_alias=zdlingest-scan:1521/zdlra:dedicated')";
36. Adding Protected Database manually
ZDLRA - in Action36 21.09.2016
Enable Real-Time Redo (Delta Push)
Restart the Database
Configure the RMAN Channel for ZDLRA
ALTER SYSTEM SET REDO_TRANSPORT_USER=ra_orcl03 SCOPE=BOTH;
[oracle@odax30 ~]$ srvctl stop database -db orcl03
[oracle@odax30 ~]$ srvctl start database -db orcl03
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' FORMAT '%d_%U' PARMS
"SBT_LIBRARY=/u01/app/oracle/product/12.1.0.2/dbhome_1/lib/libra.so,
ENV=(RA_WALLET='location=file:/u01/app/oracle/product/12.1.0.2/dbhome_1/dbs/ra_wallet
credential_alias=zdlingest-scan:1521/zdlra:dedicated')";
37. Execute Backup manually
ZDLRA - in Action37 21.09.2016
Backups are like in traditional way
Only "forever" incremental level 1 backups should be schedule
Level 0 Backup are of course always possible
It works also with old fashion syntax soft link from libra.so to libobk.so are
needed !!!
Scheduling is also free selectable, EM is recommended !
[oracle@odax30 ~]$ rman target / catalog ra_orcl02/manager@zdlingest-scan:1521/zdlra
...
connected to target database: ORCL02 (DBID=3592015831)
connected to recovery catalog database
RMAN> backup incremental level 1 database;
run {
allocate channel c1 type sbt_tape;
backup incremental level 1 database;
}
...
ORA-27211: Failed to load Media Management Library
38. ZDLRA - in Action38 21.09.2016
Recovery Capabilities
39. General statement about Recoveries of protected
Databases
ZDLRA - in Action39 21.09.2016
Recoveries of protected databases can be done with EM or manually
With EM several Recovery-Scenarios are possible
All Recovery scenarios with EM are full automated
(e.g. Creation of auxiliary Databases, etc.)
EM generates RMAN Scripts for each Scenario
with the possibility of adaption
Possibility of selection between full and point-in-time
recoveries
Duplicate Database (for standby) need some preparation
(e.g. FS-Structure (admin tree), Password-File, etc.)
40. Protected Database Recovery with EM
ZDLRA - in Action40 21.09.2016
From database Page under Availability Backup & Recovery Perform Recovery
41. Protected Database Recovery with EM
ZDLRA - in Action41 21.09.2016
Choose the needed Recovery Scenario
... You can choose an other restore
location if needed...
42. Protected Database Recovery with EM
ZDLRA - in Action42 21.09.2016
A schedule will be created
... And you can adapt the script if needed and submit the job...
43. Protected Database Recovery with EM
ZDLRA - in Action43 21.09.2016
The Job can be monitored by clicking the "View Job" button...
44. Protected Database Recovery manually
ZDLRA - in Action44 21.09.2016
Restore can be done also manually
– of course db must be mounted before...
– RMAN Client should be configured otherwise parameter are needed (e.g. ENV=...)
Or a point-in-time Recovery
run {
restore database;
recover database;
alter database open;
}
run {
set until time '08.09.2016 08:45:00';
restore database;
recover database;
alter database open resetlogs;
}
45. ZDLRA - in Action45 21.09.2016
Recovery Appliance Views and
Utilities
46. Some RA Views
ZDLRA - in Action46 21.09.2016
It is also possible to read the Recovery Appliance Information over Views from Recover
Appliace Metadata Database
View Name Description
RA_DATABASE Information about Protected Databases
RA_DB_ACCESS Describes User Accounts that have access to which protected database
RA_PROTECTION_POLICY Protection Policies defined on the Recovery Appliance
RA_DATABASE_STORAGE_USAGE Storage usage for each protected database
RA_RESTORE_RANGE Describe Restore Range for each protected database
RA_INCIDENT_LOG Describe Recovery Appliance Incidents
RA_REPLICATION_SERVER Replication Server Information (if used)
47. rastat.pl Utility
ZDLRA - in Action47 21.09.2016
rastat Utility help to evaluate the performance of Recovery Appliance
– NETBACKUP/NETRESTORE
• Measure the network performance
– ASMREAD/ASMWRITE
• Measure the ASM Disk I/O Performance
– CONTAINERREAD/CONTAINERWRITE/CONTAINERALLOC
• Measure the Container Disk I/O Performance
oracle@zdldbadm01:/opt/oracle.RecoveryAppliance/client/ [zdlra1] perl
/opt/oracle.RecoveryAppliance/client/rastat.pl --test=ASMREAD --filesize=2048M --
rasys=rasys/manager@zdlingest-scan:1521/zdlra --diskgroup=+CATALOG
RECOVERY APPLIANCE READ IO TEST FROM DISK
Disk Group: +CATALOG
2048 MB, 1.90s read IO time, .10s CPU time, 1076.40 MB/s, 5.26% CPU usage
PL/SQL procedure successfully completed.
48. dbms_ra Package
ZDLRA - in Action48 21.09.2016
dbms_ra provides administration function from inside the RA Database
Ca. 50 Procedure for different types of administration
– Start/Stop of Recovery Appliance
– Add/Delete Protected Databases
– Grants Access to specific User to enable to backup and restore
– Add/Update Protection Policies
– Manage Tape Configuration (if available)
– Add Repliaction Server (if available)
But, the utilization of Enterprise Manager is recommended !!!
50. Demo
ZDLRA - in Action50 21.09.2016
Recovery Appliance Page
Configure Protected Database
Run a Backup
Real Time Redo Transport
Restore a Protected Database
52. Conclusion
ZDLRA - in Action52 21.09.2016
ZDLRA worked in our Tests as
expected
Very good implementation of well
known Backup and Recovery
Technology on Engineered System
RPO is (near) zero with Real Time
Redo Transport
Even RTO ist much better then
existing systems
Very simple Management with EM
Risk reduced, cost reduced,
qualitiy improved
ZDLRA will find it’s
customer !
53. Questions and Answers...
Daniele Massimi – Principal Consultant
Tel. +41 58 459 50 92
daniele.massimi@trivadis.com
21.09.2016 ZDLRA - in Action53