This document discusses using data virtualization to accelerate application projects by 50%. It outlines some common problems with physical data copies, such as bottlenecks, bugs due to old data, difficulty creating subsets, and delays. The document then introduces the concept of using a data virtualization appliance to take snapshots of production data and create thin clones for development and testing environments. This allows for fast, full-sized, self-service clones that can be refreshed quickly. Use cases discussed include improved development and testing workflows, faster production support like recovery and migration, and enabling continuous business intelligence functions.

1. Accelerating Application Projects by
50% with Data as a Service
kylehailey.com kyle@delphix.com @virtdata

2.  1990 Oracle
– 90 support
– 92 Ported v6
– 93 France
– 95 Benchmarking
– 98 ST Real World Performance
 2000 Dot.Com
 2001 Quest
 2002 Oracle OEM 10g
Success!
First successful OEM design
Who is Kyle Hailey

3.  1990 Oracle
– 90 support
– 92 Ported v6
– 93 France
– 95 Benchmarking
– 98 ST Real World Performance
 2000 Dot.Com
 2001 Quest
 2002 Oracle OEM 10g
 2005 Embarcadero
– DB Optimizer
Who is Kyle Hailey

4. Who is Kyle Hailey
• 1990 Oracle 90 support
 92 Ported v6
 93 France
 95 Benchmarking
 98 ST Real World Performance
• 2000 Dot.Com
• 2001 Quest
• 2002 Oracle OEM 10g
• 2005 Embarcadero
 DB Optimizer
• 2010 Delphix
When not being a Geek
- Have a little 6 year old boy
& new baby
who take up all my time

5. • Data Constraint
• Solution
• Use Cases
In this presentation :

6. The Phoenix Project
What is the
constraint
in IT ?
IT is the factory floor of this century

7. 7
Automation
Jenkins Team City Travis
Data
Virtualizatio
n
Configurati
on Chef Puppet Ansible
Compute
Virtualizatio
n Vmware OpenStack Docker
?

8. Put your energy into the constraint
Top 5 constraints in IT
1. Dev environments setup
2. QA setup
3. Code Architecture
4. Development
5. Product management
- Gene Kim Surveyed
• 14000 companies
• 100s of CIOs

9. Flow of Features
9
1
Development
Environments
2
QA & Testing Environments
Product
Management
Features
2 2
Code Architecture
3Code Speed
4
5
Data

10. Data is the constraint
60% Projects Over Schedule
85% delayed waiting for data
Data is the Constraint
CIO Magazine Survey:
only getting worse
Gartner: Data Doomsday, by 2017 1/3rd IT in crisis

11. Copy Databases:

12. Application Development Problems
12
• Not enough resources
• Contention on shared environments
• Lack of enough environments
• Late stage bug discovery
• Faulty Data leading to bugs
• Subsets
• Synthetic data
• Old data
• Slow environment builds
• Delays
• Developers waiting
• QA slow and expensive

13. Physical Data: shared bottlenecks
Frustration Waiting

14. Physical Data: bugs because of old data
Old Unrepresentative Data

15. Physical Data: subsets hard & lead to bugs
False Negatives
False Positives
Bugs in Production

16. 16
Physical Data: Production Wall

17. Physical Data: Developers wait

18. Physical Data: limited environments

19. Physical Data : late stage bugs
0
10
20
30
40
50
60
70
1 2 3 4 5 6 7
Delay in Fixing the bug
Cost
To
Correct
Dev QA UAT
# of
bugs
found
Software
Engineering
Economics
– Barry Boehm (1981)

20. Virtual Data : Expensive Refresh
20
20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST
8 Hrs8 Hrs8 Hrs8 Hrs8 Hrs8 Hrs8 Hrs 8 Hrs

21. Copies
21
• Oracle: 8-12 copies
• Fortune 2000 : 1000s of DBs
• Staggering Storage amounts

22. • Hardware
– storage, systems, network,
– rack space, power cooling
• People
– 1000s hours per year just for DBAs
– DBAs
– SYS Admin
– Storage Admin
– Backup Admin
– Network Admin
• $10s Millions for data center modernizations
Copies require People & Time

23. Typical Architecture
Production
Instance
File system
Database

24. Typical Architecture
Production
Instance
Backup
File system
Database
File system
Database

25. Typical Architecture
Production
Instance
Reporting Backup
File system
Database
Instance
File system
Database
File system
Database

26. Typical Architecture
Production
Instance
File system
Database
Instance
File system
Database
File system
Database
File system
Database
Instance
Instance
Instance
File system
Database
File system
Database
Dev, QA, UAT Reporting Backup
Triple Tax

27. Typical Architecture
Production
Instance
File system
Database
Instance
File system
Database
File system
Database
File system
Database
Instance
Instance
Instance
File system
Database
File system
Database

28. Typical Architecture
Production
Instance
File system
Database
Instance
File system
Database
File system
Database
File system
Database
Instance
Instance
Instance
File system
Database
File system
Database

29. companies unaware

30. companies unaware
Developer or AnalystBoss, Storage Admin, DBA

31. Metrics
–Time
–Old Data
–Storage
Other
–Analysts
–Audits
–Data Center Modernization
companies unaware
"we say no, no, no until we can't say no anymore"
response when IT asked for copies of prod DB

32. • Data Constraint
• Solution
• Use Cases
In this presentation :

33. Development UATQA
99% of blocks are identical

34. Solution

35. Development QA UAT
Thin Clone

36. • EMC Symmetrix
• Netapp & EMC VNX
• Solaris ZFS
Technology Core : file system snapshots
Also check out new SSD storage such as: Pure Storage, EMC XtremIO

37. Fuel not equal car
Challenges
1. Technical
2. Bureaucracy

38. 1. Bureaucracy
Developer Asks for DB Get Access
Manager approves
DBA Request
system
Setup DB
System
Admin
Request
storage
Setup
machine
Storage
Admin
Allocate
storage
(take snapshot)

39. Why are hand offs so expensive?
1hour
1 day
9 days
1. Bureaucracy

40. 2. Technical Challenge
Database
Luns
Production Filer
Target A
Target B
Target C
snapshot
clones
InstanceInstance
InstanceInstance
InstanceInstance
InstanceInstance
Instance
Source

41. Database
LUNs
snapshot
clonesProduction Filer
Development Filer
2. Technical Challenge
Instance
Target A
Target B
Target C
InstanceInstance
InstanceInstance
InstanceInstance
Instance

42. Data Flow Optimization
42

43. 43© 2015 Delphix. All Rights Reserved. Private & Confidential.
Install Delphix on Intel hardware
• .
• .
• .
• .
• .
• Data
• .
• Binaries
• Application Stacks
• EBS
• SAP
• Flat files

44. 44© 2015 Delphix. All Rights Reserved. Private & Confidential.
Allocate Any Storage to Delphix
Allocate Storage
Any type Pure Storage + Delphix
Better Performance for
1/10 the cost

45. 45© 2015 Delphix. All Rights Reserved. Private & Confidential.
One time backup of source database
Data is
compressed
typically 1/3
size
Production
3 TB 1 TB

46. 46© 2015 Delphix. All Rights Reserved. Private & Confidential.
Incremental forever change collection
Two week time flow
Production

47. 47© 2015 Delphix. All Rights Reserved. Private & Confidential.
Clones: Fast, Free, Full
Production
Two week time flow
NFS

48. Three Physical Copies
Three Virtual Copies
Data
Virtualization
Appliance

49. Before Virtual Data
Production Dev, QA, UAT
Instance
Reporting Backup
File system
Database
Instance
File system
Database
File system
Database
File system
Database
Instance
Instance
Instance
File system
Database
File system
Database
“triple data
tax”

50. With Virtual Data
Production
Instance
Dev & QA
Instance
Reporting
Instance
Backup
Instance Instance Instance
InstanceInstance
Instance
File system
Database
Instance
Instance

51. • Problem in the Industry
• Solution
• Use Cases

52. 1. Development & QA
2. Production Support
3. Business
Use Cases

53. Development: Virtual Data
Development

54. Virtual Data: Easy
Source
Clone 1
Clone 2
Clone 3

55. Virtual Data: Parallelize
gif by Steve Karam

56. Virtual Data: Full size

57. Virtual Data: Self Service

58. Environments: almost unlimited

59. QA : Virtual Data
• Fast
• Parallel
• A/B testing

60. Physical Data : find bugs fast
Dev QA UAT
# of
bugs
found
0
10
20
30
40
50
60
70
1 2 3 4 5 6 7
Delay in Fixing the bug
Cost
To
Correct

61. Dev
QA
Instance
Prod
DVA
• Fast
• Full Size
• Run Parallel QA
• Lots of environments for projects like ERP
Upgrades
Virtual Data : Parallel
Production Time Flow

62. Virtual Data: Rewind
DVAInstance
QA
Prod
Production Time Flow

63. Virtual Data : Fast Refresh
63
20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST 20 MIN TEST
• Fast
• Full
• Fresh
• Efficient
8 Hrs8 Hrs8 Hrs8 Hrs8 Hrs8 Hrs8 Hrs 8 Hrs
20 MIN
TEST

64. Virtual Data: A/B
DVAInstance
Instance
Instance
Index 1
Index 2
Production Time Flow

65. Virtual Data: Version Control
4/30/2015 65
Dev
QA
2.1
Dev
QA
2.2
2.1 2.2
Instance
Prod
DVA Production Time Flow

66. Physical Data: Copies increase the surface area of risk !
Production

67. Virtual Data: reduce surface area further protected by masking
Production
NFS

68. 1. Development and QA
2. Production Support
3. Business
Use Cases

69. • Recovery
• Forensics
• Migration
Production Support

70. 9TB database 1TB change day : 30 days
0
10
20
30
40
50
60
70
week1
week2
week3
week4
original
Oracle
Delphix
Storage
Required
(TB)
Days

71. RPO & RTO
71
• RPO
– Any time in last 30 days
– Down to the second
• RTO
– Minutes
– Push button
0
5
10
15
week1
week2
week3
week4 original
Delphix

72. Virtual Data: Recovery
Instance
Instance
Recover VDB
Drop
Source
DVA Production Time Flow

73. Virtual Data: Forensics
Instance
Development
DVA
Source
Production Time Flow

74. Virtual Data: Development recovery
Instance
Development
DVA
Source
Development
Prod & VDB Time Flow

75. Virtual Data: Migration

76. Cloud Migration and Replication
76

77. 1. Development and QA
2. Production Support
3. Business Continuity
Use Cases

78. Business Intelligence
• Audits
• ETL
• Temporal
• Federated data
• Consolidated data

79. Production Time Flow
Virtual Data: Audit
4/30/2015 79
Instance
Prod
DVA
Live Archive
Live Archive data for years
• Archive EBS R11 before upgrade to R12
• Sarbanes-Oxley
• Dodd-Frank
• Financial Stress tests

80. Business Intelligence: ETL and Refresh Windows
1pm 10pm 8am
noon

81. Business Intelligence: batch taking too long
1pm 10pm 8am
noon
2011
2012
2013
2014
2015

82. 2012
2013
2014
2015
1pm 10pm 8am
noon
10pm 8am noon 9pm
6am 8am 10pm

83. Business Intelligence: ETL and DW Refreshes
Instance
Prod
Instance
DW & BI

84. • Collect only Changes
• Refresh in minutes
Instance
Prod
BI and DW
ETL
24x7
DVA
Virtual Data: Fast Refreshes
Time Flow

85. Modernization: Federated
Instance
Instance
Source1
Source2
Production Time Flow 1
Production Time Flow 2

86. Physical Data: Federated

87. “I looked like a hero”
Tony Young, CIO Informatica
Virtual Data: Federated

88. Virtual Data: Temporal Data

89. Virtual Data: Confidence testing

90. 1.Development & QA
– Dev throughput increase by 2x
2. Production Support
– 30 days in size of source
3. Business Continuity
– 24x7 ETL & federated cloning
Use Case Summary

91. © 2015 Delphix. All Rights Reserved. Private & Confidential. P91.© 2015 Delphix. All Rights Reserved. Private & Confidential. P91.
Shift Left
ROI
Time
Reduced
OpEx, CapEx
B
• Insurance product “about 50 days ... to about 23 days”
– Presbyterian Health
• “Can't imagine working without it”
– State of California
• Projects “12 months to 6 months.”
– New York Life

92. • Projects “12 months to 6 months.”
– New York Life
• Insurance product “about 50 days ... to about 23 days”
– Presbyterian Health
• “Can't imagine working without it”
– State of California
Virtual Data Quotes

93. • Problem: Data constraint
• Solution: Data Virtualization
Summary
Innovation
• Transformative
• Automation
• Self Service

94. Thank you!
• Kyle Hailey - Technical Evangelist (Oracle Ace, Oaktable)
– Kyle@delphix.com
– kylehailey.com
– slideshare.net/khailey
– @virtdata

95. One other thing: Performance
• Performance
95

96. Oracle 12c

97. 80GB buffer cache ?

99. 5000
Tnxs/minLatency
300
ms
1 5 10 20 30 60 100 200
with
1 5 10 20 30 60 100 200
Users

100. 200GB
Cache

101. 5000
Tnxs/minLatency
300
ms
1 5 10 20 30 60 100 200
Users
with
1 5 10 20 30 60 100 200

102. $1,000,000
1TB cache on SAN
$6,000
200GB shared cache on Delphix
Five 200GB database copies are
cached with :

103. Goal : virtualize, govern, deliver
103
• Masking: Masking
• Security: Chain of custody
• Self Service: Logins
• Developer: Versioning , branching
• Audit: Live Archive
Snap Shots
Thin Cloning
Copy Data Management
Data as a Service
31 2
2
32
EMC, Netapp, ZFS
Oracle Snap Clone, Clone DB
Actifio
Datical,
Oracle DBaaS
Delphix

104. • EMC Symmetrix
– 16 snapshots
– Write performance impact
– No snapshots of snapshots
• Netapp & EMC VNX
– 255 snapshots
• ZFS
– Compression
– Unlimited snapshots
– Snapshots of Snapshots
• DxFS
– Compression
– Unlimited snapshots
– Snapshots of Snapshots
– Shared cache in memory
Technology Core : file system snapshots
Also check out new SSD storage such as: Pure Storage, EMC XtremIO

105. Actifio
Production
InstanceInstanceInstance
Actifio
InstanceInstance Instance
TargetActifio
Instance
Target

106. Oracle Snap Clone
ZFSSA
or
NetApp
Instance
TargetEM 12c
Instance
Target
Production
InstanceInstanceInstance

107. Oracle Snap Clone
Production
InstanceInstanceInstance
Data Guard
InstanceInstanceInstance
ZFSSA
or
NetApp
Instance
TargetEM 12c
Instance
Target

108. Oracle Snap Clone
Production
InstanceInstanceInstance
Solaris
ZFS
Instance
TargetData Guard
Instance
Instance
Target
Any
storage
EM 12c

109. Incremental forever collect changes
Production
InstanceInstanceInstance
Time Flow
Changes
Instance
NFS
Target
Instance
Target

110. Data virtualization
• Fast becoming the new norm
• Used by Over 100 of Fortune 500
• Enables DevOps

111. 111
Automation
Jenkins Team City Travis
Data
Virtualizatio
n Delphix Open ZFS Flocker
Configurati
on
Managemen
t
Chef Puppet Ansible
Compute
Virtualizatio
n VMware Vagrant Docker AWS OpenStack

112. 112
Jenkins, Team City, Travis
Open Stack, Vagrant, Docker
Chef, Puppet, Ansible
Delphix
DevOps : Automation + Culture

113. Snapshot 1 - full backup
Jonathan Lewis © 2013
Virtual DB
113 / 30
a b c d e f g h i

114. Snapshot 2 - incremental
Jonathan Lewis © 2013
b' c'
a b c d e f g h i

115. Snapshot 2 - apply
Jonathan Lewis © 2013
a b c d e f g h ib' c'

116. Snapshot 1 – drop
Jonathan Lewis © 2013
b' c'a d e f g h i

117. Creating a VDB
Jonathan Lewis © 2013
b' c'a d e f g h i
My vDB
(filesystem)
Your vDB
(filesystem)
b' c'a d e f g h i

118. Modify a vDB
Jonathan Lewis
© 2013
b' c'a d e f g h i
My vDB
(filesystem)
Your vDB
(filesystem)
i’b' c'a d e f g h ib' c'a d e f g h i

119. What is DevOps ?
119
• Not Tools (required)
• Not a Process (not standardized yet)
• Not Culture (critical)
DevOps is a Goal

120. DevOps Goal :
120
Fast flow of features
from development
to IT operations
to the customers
- Gene Kim

121. The Problem: Flow of Features
121
Features Customer

122. The Goal : eliminate the constraint
Improvement
not made
at the constraint
is an illusion
Theory of Constraints

123. Factory floor

124. Factory floor
constraint

125. Factory floor
constraint
Tuning here
Stock piling

126. Factory floor
constraint
Tuning here
Starvation

127. Factory floor
constraint
Goal:
• find constraint
• optimize it

128. A database refresh in 15 minutes?
That is mind blowing!
Delphix nailed it for us.
- Matt Lawrence , Sr Director Wind River (Intel)
Took 3 weeks to build a dev env
now with Delphix takes less than a day
the db part is less than 15 minutes
- Marty Boos , Stubhub (Ebay)
Delphix goes beyond storage
Delphix so much more than
We thought it was
-Michael Brow State of Colorado

129. Worth investing on this product
the technology is strong and
value prop is high
- Deloitte
I'm convinced about Delphix's
technology Delphix can really
increase the quality of Dev / QA
- Oaktable Member
Delphix allows us to move fast and setup database copies in seconds
Delphix is powerful and allowed us to scale from 2 projects to 11
We need Delphix to scale our agile environment
– Tim Campos, CIO, Facebook