1. Database Virtualization
and
Instant Cloning
Kyle Hailey
http://dboptimizer.com

2. Database Cloning Challenge
Business want data now.
Business don’t understand DBAs.
Databases getting bigger & harder to copy.
Developers want more copies.
Reporting wants more copies.
Everyone has storage constraints.
If you can’t satisfy the business
demands your process is broken.

3. Two Parts
I. Cloning Technology
II. Accelerate your business

4. Part I : Cloning Technology
3. Virtual2. Thin Provision1. Physical

5. =
database
1. Physical Cloning

6. Problem
Developers
QA and UAT
Reports
First
copy
Production
• CERN - European Organization for Nuclear
Research
• 145 TB database
• 75 TB growth each year
• Dozens of developers want copies.

7. workarounds
Developers
QA and UAT
Reports
Shared
Sub set copy
Production
Many copies

8. Physical Clones
Database Subsets
Shared Databases

9. Subsets

10. The Production
‘Wall’
Classic problem is that queries that
run fast on subsets hit the wall in
production.
Developers are unable to test against
all data

11. Shared Full

12. Shared access = Poor Productivity
Developers and
tester get frustrated
Databases become old
and unrepresentative
of production.
Requires complex
scheduling and
management

13. Never enough environments
Average customer makes 12 copies of production
- Charles Garry, Database Product Manager Oracle

14. Physical Copies – Time & Space
consuming
Time consuming
Time to make copies, days to weeks
RMAN backup, archive logs, copy data over, recover
Meetings , days to weeks
System, Storage ,Database ,Network Admins
, manager coordination
Space consuming
12 devs x 10TB production = 120TB
20 report DBs x 40 TB = 800TB
Slow Clones mean bottlenecks

16. Setup Develop

17. Setup

18. Setup Develop
QA

19. $40M
$75M
$850M
$27,000M
Storage
IT
Develop
Business

21. ERP Project Failures 2011
• NYC CityTime : delays $63 M => $760 M
• Montclair Uni: delays sues PeopleSoft
• Idaho : delays ERP cost millions
Standish : IT Project Failure Rate
1994 1996 1998 2000 2002 2004 2009
31% 40% 28% 23% 15% 18% 24%
★http://www.galorath.com/wp/software-project-failure-costs-billions-better-estimation-planning-can-
help.php*http://www.pcworld.com/article/246647/10_biggest_erp_software_failures_of_2011.html

23. Clone 1 Clone 3
99% of blocks are Identical
Clone 2

25. 2. Thin Provision Cloning

26. Clone 1 Clone 2 Clone 3
Thin Provision

27. 2. Thin Provision Cloning
Requires Cornerstone Technology

28. I. clonedb
II. Copy on Write Snapshots
a) EMC BCV
b) EMC SRDF or Recover Point
c) Vmware
III. Allocate on Write
a) Netapp (EMC VNX)
b) ZFS
c) DxFS
2. Thin Provision Cloning

29. RMAN
backup
dNFS
sparse file
I. clonedb

30. RMAN
backup
dNFS
sparse file
I. clonedb

31. I. CloneDB
• dNFS 11.2.0.2+
– cd $ORACLE_HOME/rdbms/lib
– make -f ins_rdbms.mk dnfs_on
• Clonedb.pl initSOURCE.ora output.sql
– MASTER_COPY_DIR="/rman_backup”
– CLONE_FILE_CREATE_DEST="/nfs_mount”
– CLONEDB_NAME="clone"
• sqlplus / as sysdba @output.sql
– startup nomount PFILE=initclone.ora
– Create control file backup location
– dbms_dnfs.clonedb_renamefile
('/backup/file.dbf' , '/clone/file.dbf');
– alter database open resetlogs;
Tim Hall
www.oracle-base.com/articles/11g/clonedb-11gr2.php

32. I. CloneDB
RMAN
physical
Target A NFS Server
Clone 1
Clone 2
Read only Clone 3
Target B
Clone 1
Clone 2
Clone 3

33. I. clonedb
II. Copy on Write + snapshots
a) EMC BCV
b) EMC SRDF or Recover Point
c) Vmware
III. Allocate on Write
a) Netapp (EMC VNX)
b) ZFS
c) DxFS
2. Thin Provision Cloning

34. D
Active
File
System
Snapshot
CBA
File System constant - EMC
II. Copy on Write a) EMC snapshots

35. D
Active
File
System
Snapshot
DCBA
Changes written in place - EMC
Changes to new area - VMware
II. Copy on Write a) EMC snapshots

36. • Goal: backup
– Create BCV , take snaps of BCV
– Zone and mask LUN to target host
– Full copy of disk, snapshot, recover
• Snapshots
– 16 snapshots
– Zero or one snapshots of snapshots
• “Golden Copy”
– EMC uses a save area, area for changes to the snapshot
– Initial snapshot has to stay
II. Copy on Write a) EMC BCV

37. Non-Prod FilerEMC Filer
Production
Database
Database
Luns
Target A
Target B
Target C
Clone 1
Clone 2
Clone 3
Clone 4
Snapshots
Max 16
a) EMC
BCVIII. Allocate on Write
BCV
Day 1
snapshot
Snapshot
or break
Day 2
Timefinder needed for snapshots
on across multiple LUNs

38. • SRDF – Symetrix Remote Data Facility
– Stream Changes to a remote machine
• Recover Point
– Capture changes on wire
– Split changes send to remote site
II. Copy on Write b) EMC SRDF

39. Non-Prod FilerEMC Filer
Production
Database
Database
Luns
Target A
Target B
Target C
Clone 1
Clone 2
Clone 3
Clone 4
File system level
SRDF or
Recover Point
Snapshot
day 1
Snapshot
day 1
Snapshot
day 2
Snapshot
day 2
II. Copy on Write b) EMC SRDF

40. D
Active
File
System
Snapshot
DCBA
Changes written to a new location
Changes to new area - VMware
II. Copy on Write, c) VMware

41. II Copy on Write c) VMware
Data Director : Linked Clones
• Not support for Oracle databases
• Golden Copy : rebuild after 32 snapshots
• x86 host databases only
• Performance issues
– “Having several linked clones can affect the performance of the source database and the
performance of the linked clones.”
http://bit.ly/QOXbyE (on http://pubs.vmware.com )
– “If you are focused on performance, you should prefer a full clone over a linked clone.”
http://www.vmware.com/support/ws5/doc/ws_clone_typeofclone.html
– Performance worse with more snapshots
– Performance worse with more concurrent users

42. I. clonedb
II. Copy on Write
a) EMC BCV
b) EMC SRDF or Recover Point
c) Vmware
III. Allocate on Write
a) ZFS
b) Netapp (EMC VNX)
c) DxFS
2. Thin Provision Cloning

43. Data Blocks
root
a.ZFS
b.Netapp (EMC VNX)
c. Dxfs
III. Allocate on Write

44. a) ZFS
• 1 disk = 1 filesystem
• ~1990: N disks = 1 FS
• 2001: ZFS starts
• 2005: ZFS ships
• 2008: ZFS storage appliance ships
• 2010: Delphix heads ZFS open source
III. Allocate on Write

45. FS vs. ZFS
• FS per Volume
• FS limited bandwidth
• Storage stranded
• Many FS in a pool
• Grow automatically
• All bandwidth
Storage PoolVolume
FS
Volume
FS
Volume
FS ZFS ZFS ZFS
3a) ZFSIII. Allocate on Write

46. a) ZFS
Snapshot root
Live
root
Delphix Proprietary and Confidential
Zil
Intent Log
III. Allocate on Write

47. ZFS Appliance + RMAN
1. ZFS Appliance
– Project db_master
– Project db_clone
– 4 file systems: datafile, redo, archive, alerts
2. Source Database
– NFS Mount ZFS Appliance
– Backup with RMAN
3. ZFS Appliance
• Select db_master
• Snapshots
• Then each filesystem on db_master clone it onto db_clone
4. Target Host
– NFS Mount db_clone
– recover clone
cloning-solution-353626.pdf
a) ZFSIII. Allocate on Write

48. Oracle ZFS Appliance + RMAN
1. physical
ZFS Storage Appliance
RMAN
copy
RMAN Copy
to
NFS mount
Target A
Clone 1
Clone 1
Snapshot
NFS
a) ZFSIII. Allocate on Write
cloning-solution-353626.pdf

49. Oracle ZFS Appliance + Dataguard
Dataguard
ZFS Storage Appliance
datafiles
NFS
Target A
Clone 1
Clone 1
Snapshot
NFS
a) ZFSIII. Allocate on Write
Datafiles via
NFS
maa-db-clone-szfssa-172997.pdf

50. NetApp FilerNetApp Filer
Production
Database
Database
Luns
snapshot
Target A
Target B
Target C
Clone 1
Clone 2
Clone 3
Clone 4
File system level
snapshot
clones
Snap mirror
Snapshot
Manager for
Oracle
Flexclone
b) NetappIII. Allocate on Write
Requires a
database

51. NetApp FilerNetApp Filer
Production
Database
Database
Luns
snapshot
Target A
Target B
Clone 1
Clone 2
snapshot
snapshot
snapshot
b) NetappIII. Allocate on Write

52. NetApp FilerNetApp Filer
Physical
Database
Database
Luns
snapshot
Target A
Target B
Clone 1
Clone 2
snapshot
snapshot
snapshot
b) NetappIII. Allocate on Write

53. • Compression
• typically 3x
• Block sharing
• DxFS optimized for databases
• Write optimizations
• Space allocation and destroy
• Shared blocks in memory
c) DxFSIII. Allocate on Write

54. I. clonedb
II. Copy on Write Snapshots
a) EMC BCV
b) EMC SRDF or Recover Point
c) Vmware
III. Allocate on Write
a) Netapp (EMC VNX)
b) ZFS
c) DxFS
2. Thin Provision Cloning

55. • Clonedb
• EMC BCV
• Netapp
• EMC SRDF
• EMC VNX
2. Review: Thin Provision Cloning

56. 2. Thin Provision Cloning

58. 2. Thin Provision Cloning

60. 3. Database Virtualization
• Oracle SMU
• Delphix

61. Virtualization
Layer
Virtualization
SMU

62. a) Oracle 12c SMU
Oracle Snap Management Utility for ZFS Appliance
• Requires ZFS Appliance
• Supports Linux , Solaris 10+, Windows 2008+
• GUI
– snapshot source databases
– provision virtual databases
3. Database Virtualization

64. Virtualization Layer
x86 hardware
Allocate
Storage
Any type
SMU
Choose your virtualization Layer:
• Delphix and Oracle SMU automated out of box
• Netapp and EMC SRDF/Recover Point require massive scripting
• Delphix and possibly SMU share blocks in memory
ZFS Storage Appliance

65. One time backup of source database
Database
Production
Instance
File system
RMAN APIs

66. Delphix Compress Data
Database
Production
Instance
File system
Data is
compressed
typically 1/3
size

67. Incremental forever change collection
Database
Production
Instance
File system
Changes are collected
automatically forever
Data older than retention
widow freed

68. Typical Architecture
Database
File system
Production
Instance
Database
File system
Development
Instance
Database
File system
QA
Instance
Database
UAT
Instance
File system

69. Clones share duplicate blocks
Development
Instance
Database
Production
Instance
File system
vDatabase
QA
Instance
UAT
Instance
vDatabase vDatabaseNFS
Source Database Clone Copies of Source Database
Fiber Channel

70. Part II: Virtualization Case Studies
1. Development
2. QA
3. Recovery

71. 1. Development Acceleration

72. Database
Virtualization
layer
3 clones of same source database 3 virtual clones of same source database
Virtualization layer orchestrates I/O
access between database and storage

73. Source Timeflow
Source Timeflow
Source Database
Target Host
Virtual Database
NFS
Fiber
Fiber
RMAN over
TCP

74. Source
Fresh
VDB

75. Source
VDB
VDB
VDB
Frequent

76. Fresh data
Full clones
Frequent (many)
fast

78. Self Service

79. Dev
v2.6 v2.6v2.6
QA UAT
v2.6
Production
v2.6 v2.6v2.6
v2.7
v2.6 v2.6v2.6
v2.8
Source Control for the database data
v2.6
v2.6
v2.6
v2.6
v2.6
v2.7
v2.6
v2.7
v2.6
v2.8
v2.6
v2.8

80. Dev
Prod
2.6

81. Dev
QA
Prod
2.6

82. Dev
QA
UAT
Prod
2.6

83. Dev
QA
UAT
Prod
Dev
QA
UAT
2.6
2.7

84. Dev
QA
UAT
Prod
Dev
QA
UAT
2.6
2.7
Dev
QA
UAT
2.8

85. Dev
QA
UAT
Prod
Dev
QA
UAT
2.6
2.7
Dev
QA
UAT
2.8
Data Control = Source Control for the Database

86. Collaborative Provisioning

88. VDB2
Source2
Source3
Source1
VDB3
VDB1
Collaborative sources

89. “I looked like a hero”
Tony Young, CIO Informatica

90. 1: Review of Development
• Provision , Self Service
– Fast
– Full
– Fresh
– Frequent (Many)
• Source control for the database
• Federated cloning

91. 2. Rapid QA

92. dSource
Pass to QA
Developer VDB
QA VDB

93. Source
Punt to Dev
Developer VDB
QA VDB
Developer VDB

94. dSource
Forensic Analysis
VDB

95. dSource
Upgrades, Patches, RAT
VDB

96. A/B Testing
• Production vs Virtual
– invisible index on Prod
– Creating index on virtual
• Flashback vs Virtual
• Confidence testing
– change reporting SQL
– compare new code over last 30 days

97. 2: review QA
• Fork copies off of Dev to QA
• QA fork copies back to Dev
• Instant replay
– performance A/B
– Upgrade, patching
– Forensic analysis of Prod

98. 3. Backup and Recovery

99. MONSUN WEDTUE THU SATFRI MONSUN TUE
MONSUN WEDTUE THU SATFRI MO
N
SUN TUE
DataProcessing
Required
Current Volume
of Work

101. Source
Rollback / Time machine
VDB

102. Source
Rollback / Time machine
VDB
VDB Rolled back

103. Source
Recovery
VDB
V2P

104. 3: review Backup
• Backup 50 days in size of 1 copy
• logical recovery on prod, dev
• Restore virtual to production

105. Review Part II: Data Virtualization
1. Development: Full, Fresh, Fast, Data SCCS = SDCS
2. QA : forking, replay, forensics
3. Recovery: 50:1, time machine, V2P
Review Part I: Technology
1. Full Clones
2. Thin Provision : dbclone, EMC, Netapp
3. Database Virtualization : SMU, Delphix

106. over 10 times
"perhaps the single largest storage consolidation
opportunity history“

110. Oracle 12c

111. 80MB buffer cache ?

112. 5000
Tnxs/minLatency
300
ms
1 5 10 20 30 60 100 200
with
1 5 10 20 30 60 100 200
Users

113. 200GB
Cache

114. 5000
Tnxs/minLatency
300
ms
1 5 10 20 30 60 100 200
Users
1 5 10 20 30 60 100 200

115. 200GB
Cache

116. 8000
Tnxs/minLatency
600
ms
1 5 10 20 30 60 100 200
Users
1 5 10 20 30 60 100 200

117. Database Virtualization

118. Matrix of features
CloneDB ZFS
Appliance
Delphix Data
Director
NetApp EMC
Time Flow No Yes Yes No Yes No
Hardware
Agnostic
Yes No Yes Yes No No
Snapshots No Unlimited Unlimited 31 255 16
(96 read only)
Snapshots of
snapshots
No Unlimited Unlimited 30 255 1
Automated
Snapshots
No No Yes No Yes No
Automated
Provisioning
No No Yes No No No
Any O/S Yes Yes Yes No
x86 only
Yes Yes
Max size None None None ? 16-
100TB
?

119. Conclusion : Enterprise Solutions
• VMware Data Director
• EMC Timefinder
– offer limited ability to benefit from cloning
• Clonedb ***
– fast easy way to create many clones of the same copy
– limited to 11.2.0.2 and systems with sparse file system capability
– suffers the golden image problem and performance
• NetApp Flexclone, Snap Manager for Oracle
– offers a rolling solution
– limited database awareness
– file system clones
– limited snapshots
– Vendor lock-in
• Oracle ZFS Appliance
– Vendor Lock-in
• Delphix
– Agility : Automation, unlimited snapshots, clones, multi-database

120. Appendix
• CloneDB
– http://www.oracle-base.com/articles/11g/clonedb-11gr2.php
• ZFS
– http://hub.opensolaris.org/bin/download/Community+Group+zfs/docs/zfslast.pdf
• ZFS Appliance
– ZFS setup for Oracle http://www.oracle.com/technetwork/server-storage/solaris/config-solaris-zfs-wp-167894.pdf
– using RMAN http://www.oracle.com/technetwork/articles/systems-hardware-architecture/cloning-solution-353626.pdf
– using dataguard http://www.oracle.com/technetwork/database/features/availability/maa-db-clone-szfssa-172997.pdf
• Data Director
– http://pubs.vmware.com/datadirector-25/topic/com.vmware.ICbase/PDF/vfabric-data-director-25-administration-guide.pdf
p107 linked clone not support for Oracle
• EMC
– https://community.emc.com/servlet/JiveServlet/previewBody/11789-102-1-45992/h8728-snapsure-oracle-dnfs-wp.pdf
– http://www.centroid.com/knowledgebase/blog/cloning-oracle-databases-with-emc-snapview-and-recoverpoint
– http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:39960485246162 BCV and SRDF
• NetApp
– http://www.redbooks.ibm.com/redpapers/pdfs/redp4133.pdf
– OEM for Netapp http://www.youtube.com/watch?v=J7fnfLS5Dxg&feature=youtu.be
• Oracle SMU
– http://www.oracle.com/us/products/servers-storage/storage/nas/smu-bus-wp-final-1857065.pdf
– http://www.oracle.com/us/products/servers-storage/storage/nas/snap/smu-datasheet-1903178.pdf
– http://www.oracle.com/us/products/servers-storage/storage/nas/snap/overview/index.html
• Oracle OEM Snapshot Manager for Netapp
– http://www.youtube.com/watch?v=J7fnfLS5Dxg&feature=youtu.be
– Demo starts at 3:10 goes to 19:43

121. Other
• Can’t run multiple clones on same host with
ASM
https://forums.oracle.com/forums/thread.jsp
a?messageID=1810251
• EMC VNX
http://virtualgeek.typepad.com/virtual_geek/
2012/05/vnx-inyo-is-going-to-blow-some-
minds.html

122. NetApp Limits
Controller Size Limit
32 bit controllers 16TB
FAS3140/FAS3040/FAS3050 40TB
FAS3160/FAS3070 50TB
FAS6040/FAS3170 70TB
FAS6080 100TB
All sources have to be in the same aggregate to be
snapshot together.
Limit of 255 snapshots
snaps are limited to the same aggregate (storage pool)
Aggregates have size limits depending on controller

123. b) NetappIII. Allocate on Write

124. b) NetappIII. Allocate on Write

125. b) NetappIII. Allocate on Write

126. Netapp
vs
EMC

127. Production
Initial
Incremental Incremental
Redo
Clones
Redo
a) ZFSIII. Allocate on Write
Goal - How Can we do it?

128. Data Director
CloneDB NetApp
EMC
Delphix
SMU
ZFS
Flawed
Limited Use
Manual
low touch: virtualization
Automated (easy)
Manual (Hard)
FullFunctionality
Doesn’twork

129. Application Life Cycle Management
Number of required clones change during Project and Extended Project Life Cycle
Projects with phased Go-Lives can have Implementation, Handover ‘Cut-Over’ and
Operations ‘RUN’ all running concurrently.

130. Development & maintenance is hard
User
Acceptance
Test
TestDevelopment
Production
Training &
Education
Production
Support
New release / upgrades lifecycle…
Bespoke, enhancements, configuration changes etc…
Production Support lifecycle…
Internal bug fixes, 3rd Party Patches, Data fixes etc..
User
Acceptance
Test
TestDevelopment
Training &
Education
Production
Support
Multiple Clones used by Production Support
Analysts, DBA team and for internal training
DBA/
Note: All environments periodically refreshed from production

131. Database’s not just used by developers
Data Migration Team
Data Mapping & Cleansing
Development of ETL
(Extraction, Transformation
& Load) Routines
…
User
Acceptance
Test
User
Acceptance
Test
System
Validation
Testing
Data
MigrationData
MigrationData
Migration
Technical Teams
System Integration
Enterprise Scheduling
Automated Testing
Performance Testing
…
Application Functional
Specialists
Development of System
Set-up
Business Process
Configuration
Sandbox
…
Business Process Owners
Application Functional
Process validation
CRP (Conference Room
Pilot)
…
User
Acceptance
Test
User
Acceptance
Test
User
Acceptance
Test
Automated
TestingAutomated
TestingAutomated
Testing

132. Source Timeflow
Source Timeflow
Source Database
Target Host
Virtual Database
NFS
Fiber
Fiber
RMAN over
TCP