“Accelerating DevOps Using Data Virtualization” at the Collaborate 2016 conference in Las Vegas. It discusses the inevitability of data virtualization and its many use cases.

1. © 2015 Delphix. All Rights Reserved
Session ID:
Prepared by:
Accelera'ng DevOps
Using Data Virtualiza'on
and Masking
How it all Works:
Some Internals and Use Cases
1227
@TimothyJGorman
Tim Gorman
Technical Consultant
Delphix

2. © 2015 Delphix. All Rights Reserved
For almost 25 years, I’ve been an Oracle
developer and DBA
As a developer, the job is to create new
funcMonality and fix bugs
As a DBA, the job is to create databases,
protect them, and keep applicaMons running
opMmally
This presentaMon is about all these things,
but mostly it is about change

3. © 2015 Delphix. All Rights Reserved
Data virtualizaMon (Data As A Service or
DaaS) is fast becoming the new norm
Nobody can jusMfy provisioning non-prod
environments for DEV/QA/etc the way we
did up to now
And supporMng DevOps beYer is the way
going forward

4. © 2015 Delphix. All Rights Reserved
Introducing DevOps
• IntegraMng development, tesMng, and operaMons to
increase delivery frequency
– Faster new features to market
– Lower lead Mme for fixes
– Lower rate of failure
• Geng there requires…
– End-to-end integraMon of so^ware development, tesMng,
and operaMons
– Cultural reorientaMon toward cooperaMon and empathy
– Tools and processes
4

5. © 2015 Delphix. All Rights Reserved
Introducing DevOps
• IntegraMng development, tesMng, and operaMons to
increase delivery frequency
– Goal: Faster new features to market
– Lower lead Mme for fixes
– Lower rate of failure
• Geng there requires…
– End-to-end integraMon of so^ware development, tesMng,
and operaMons
– Cultural reorientaMon toward cooperaMon and empathy
– Tools and processes
5

6. © 2015 Delphix. All Rights Reserved
Introducing DevOps
• IntegraMng development, tesMng, and operaMons to
increase delivery frequency
– Goal: Faster new features to market
– Goal: Lower lead Mme for fixes
– Lower rate of failure
• Geng there requires…
– End-to-end integraMon of so^ware development, tesMng,
and operaMons
– Cultural reorientaMon toward cooperaMon and empathy
– Tools and processes
6

7. © 2015 Delphix. All Rights Reserved
Introducing DevOps
• IntegraMng development, tesMng, and operaMons to
increase delivery frequency
– Goal: Faster new features to market
– Goal: Lower lead Mme for fixes
– Goal: Lower rate of failure
• Geng there requires…
– End-to-end integraMon of so^ware development, tesMng,
and operaMons
– Cultural reorientaMon toward cooperaMon and empathy
– Tools and processes
7

8. © 2015 Delphix. All Rights Reserved
Introducing DevOps
• IntegraMng development, tesMng, and operaMons to
increase delivery frequency
– Goal: Faster new features to market
– Goal: Lower lead Mme for fixes
– Goal: Lower rate of failure
• Geng there requires…
– End-to-end integraMon of so^ware development, tesMng,
and operaMons
– Cultural reorientaMon toward cooperaMon and empathy
– Tools and processes
8

9. © 2015 Delphix. All Rights Reserved
Introducing DevOps
• IntegraMng development, tesMng, and operaMons to
increase delivery frequency
– Goal: Faster new features to market
– Goal: Lower lead Mme for fixes
– Goal: Lower rate of failure
• Geng there requires…
– End-to-end integraMon of so^ware development,
funcMonal and performance tesMng, and operaMons
– Cultural reorientaMon toward cooperaMon and empathy
– Tools and processes
9

10. © 2015 Delphix. All Rights Reserved
Introducing DevOps
• IntegraMng development, tesMng, and operaMons to
increase delivery frequency
– Goal: Faster new features to market
– Goal: Lower lead Mme for fixes
– Goal: Lower rate of failure
• Geng there requires…
– End-to-end integraMon of so^ware development,
funcMonal and performance tesMng, and operaMons
– Cultural reorientaMon toward cooperaMon and empathy
– Tools and processes
10

11. © 2015 Delphix. All Rights Reserved
Introducing DevOps
• IntegraMng development, tesMng, and operaMons to
increase delivery frequency
– Goal: Faster new features to market
– Goal: Lower lead Mme for fixes
– Goal: Lower rate of failure
• Geng there requires…
– End-to-end integraMon of so^ware development,
funcMonal and performance tesMng, and operaMons
– Cultural reorientaMon toward cooperaMon and empathy
– Tools and processes
11

12. © 2015 Delphix. All Rights Reserved
Introducing DevOps
• IntegraMng development, tesMng, and operaMons to
increase delivery frequency
– Goal: Faster new features to market
– Goal: Lower lead Mme for fixes
– Goal: Lower rate of failure
• Geng there requires…
– End-to-end integraMon of so^ware development,
funcMonal and performance tesMng, and operaMons
– Cultural reorientaMon toward cooperaMon and empathy
– Tools and processes
12

13. © 2015 Delphix. All Rights Reserved
Introducing DevOps
Using DevOps to achieve ConMnuous Delivery takes
more than sheer will
It takes new tools and new processes
And breaking down silos built up over the years
It is not the way that Development, QA/TesMng, and
OperaMons have done things before
13

14. © 2015 Delphix. All Rights Reserved
Introducing DevOps
Using DevOps to achieve ConMnuous Delivery takes
more than sheer will
It takes new tools and new processes
And breaking down silos built up over the years
It is not the way that Development, QA/TesMng, and
OperaMons have done things before
14

15. © 2015 Delphix. All Rights Reserved
Introducing DevOps
Using DevOps to achieve ConMnuous Delivery takes
more than sheer will
It takes new tools and new processes
And breaking down silos built up over the years
It is not the way that Development, QA/TesMng, and
OperaMons have done things before
15

16. © 2015 Delphix. All Rights Reserved
Introducing DevOps
Using DevOps to achieve ConMnuous Delivery takes
more than sheer will
It takes new tools and new processes
And breaking down silos built up over the years
It is not the way that Development, QA/TesMng, and
OperaMons have done things previously
16

17. © 2015 Delphix. All Rights Reserved
Ops: “But we just finished building these silos!”
17

18. © 2015 Delphix. All Rights Reserved
Ops: “But we just finished building these silos!”
18
Regulatory compliance –
separaMon of duMes

19. © 2015 Delphix. All Rights Reserved
Ops: “But we just finished building these silos!”
19
Regulatory compliance –
separaMon of duMes
System complexity –
administer by component

20. © 2015 Delphix. All Rights Reserved
Ops: “But we just finished building these silos!”
20
Regulatory compliance –
separaMon of duMes
Data security –
least privileges
System complexity –
administer by component

21. © 2015 Delphix. All Rights Reserved
Ops: “But we just finished building these silos!”
21
Regulatory compliance –
separaMon of duMes
Data security –
least privileges
System complexity –
administer by component
From people to processes –
new tools are the soluMon

22. © 2015 Delphix. All Rights Reserved
Introducing DevOps
• I work to deploy a tool in organizaMons just beginning to
embrace DevOps
– SomeMmes these organizaMons don’t know they are
• But they are… yes they are…
• If the challenges to deployment were merely technical
– It would take two hours, tops
• But the biggest challenges are cultural and poliMcal
– So many of my deployments take months
• The good news is: only the first deployment is that painful
22

23. © 2015 Delphix. All Rights Reserved
Introducing DevOps
• I work to deploy a tool in organizaMons just beginning to
embrace DevOps
– SomeMmes these organizaMons don’t know they are embracing
anything
• But they are… yes they are…
• If the challenges to deployment were merely technical
– It would take two hours, tops
• But the biggest challenges are cultural and poliMcal
– So many of my deployments take months
• The good news is: only the first deployment is that painful
23

24. © 2015 Delphix. All Rights Reserved
Introducing DevOps
• I work to deploy a tool in organizaMons just beginning to
embrace DevOps
– SomeMmes these organizaMons don’t know they are embracing
anything
• But they are… yes they are…
• If the challenges to deployment were merely technical
– It would take two hours, tops
• But the biggest challenges are cultural and poliMcal
– So many of my deployments take months
• The good news is: only the first deployment is that painful
24

25. © 2015 Delphix. All Rights Reserved
Introducing DevOps
• I work to deploy a tool in organizaMons just beginning to
embrace DevOps
– SomeMmes these organizaMons don’t know they are embracing
anything
• But they are… yes they are…
• If the challenges to deployment were merely technical
– It would take two hours, tops
• But the biggest challenges are cultural and poliMcal
– So many of my deployments take months
• The good news is: only the first deployment is that painful
25

26. © 2015 Delphix. All Rights Reserved
Introducing DevOps
• I work to deploy a tool in organizaMons just beginning to
embrace DevOps
– SomeMmes these organizaMons don’t know they are embracing
anything
• But they are… yes they are…
• If the challenges to deployment were merely technical
– It would take two hours, tops
• But the biggest challenges are cultural and poliMcal
– So many of my deployments take months
• The good news is: only the first deployment is that painful
26

27. © 2015 Delphix. All Rights Reserved
Introducing DevOps
• I work to deploy a tool in organizaMons just beginning to
embrace DevOps
– SomeMmes these organizaMons don’t know they are embracing
anything
• But they are… yes they are…
• If the challenges to deployment were merely technical
– It would take two hours, tops
• But the biggest challenges are cultural and poliMcal
– So many of my deployments take months
• The good news is: only the first deployment is that painful
27

28. © 2015 Delphix. All Rights Reserved
Introducing DevOps
• I work to deploy a tool in organizaMons just beginning to
embrace DevOps
– SomeMmes these organizaMons don’t know they are embracing
anything
• But they are… yes they are…
• If the challenges to deployment were merely technical
– It would take two hours, tops
• But the biggest challenges are cultural and poliMcal
– So many of my deployments take months
• The good news is: only the first deployment is that painfu
28

29. © 2015 Delphix. All Rights Reserved
Introducing DevOps
• I work to deploy a tool in organizaMons just beginning to
embrace DevOps
– SomeMmes these organizaMons don’t know they are embracing
anything
• But they are… yes they are…
• If the challenges to deployment were merely technical
– It would take two hours, tops
• But the biggest challenges are cultural and poliMcal
– So many of my deployments take months
• The good news is: only the first deployment is that painful
29

30. © 2015 Delphix. All Rights Reserved
What is data virtualiza'on?
• Server virtualiza'on has had an extraordinary impact on IT
– ConverMng hardware into so^ware
– Controlling virtualized hardware through orchestraMon scripts
• But server virtualizaMon is not a complete soluMon
– Virtualizing CPU, memory (RAM), and network
– But what about storage???
• From a high-level perspecMve, server virtualizaMon has caused
the white-hot storage market to super-nova
– So easy to create virtual machines, and storage is cheap
• But is storage cheap enough?
• Data needs to be virtualized, just like servers
30

31. © 2015 Delphix. All Rights Reserved
What is data virtualiza'on?
• Server virtualizaMon has had an extraordinary impact on IT
– ConverMng hardware into so^ware
– Controlling virtualized hardware through orchestraMon scripts
• But server virtualizaMon is not a complete soluMon
– Virtualizing CPU, memory (RAM), and network
– But what about storage???
• From a high-level perspecMve, server virtualizaMon has caused
the white-hot storage market to super-nova
– So easy to create virtual machines, and storage is cheap
• But is storage cheap enough?
• Data needs to be virtualized, just like servers
31

32. © 2015 Delphix. All Rights Reserved
What is data virtualiza'on?
• Server virtualizaMon has had an extraordinary impact on IT
– ConverMng hardware into so^ware
– Controlling virtualized hardware through orchestraMon scripts
• But server virtualizaMon is not a complete soluMon
– Virtualizing CPU, memory (RAM), and network
– But what about storage???
• From a high-level perspecMve, server virtualizaMon has caused
the white-hot storage industry to super-nova
– So easy to create virtual machines, and storage is cheap
• But is storage cheap enough?
• Data needs to be virtualized, just like servers
32

33. © 2015 Delphix. All Rights Reserved
What is data virtualiza'on?
• Server virtualizaMon has had an extraordinary impact on IT
– ConverMng hardware into so^ware
– Controlling virtualized hardware through orchestraMon scripts
• But server virtualizaMon is not a complete soluMon
– Virtualizing CPU, memory (RAM), and network
– But what about storage???
• From a high-level perspecMve, server virtualizaMon has caused
the white-hot storage industry to super-nova
– So easy to create virtual machines, and storage is cheap
• But is storage cheap enough?
• Data needs to be virtualized, just like servers
33

34. © 2015 Delphix. All Rights Reserved
What is data virtualiza'on?
• Server virtualizaMon has had an extraordinary impact on IT
– ConverMng hardware into so^ware
– Controlling virtualized hardware through orchestraMon scripts
• But server virtualizaMon is not a complete soluMon
– Virtualizing CPU, memory (RAM), and network
– But what about storage???
• From a high-level perspecMve, server virtualizaMon has caused
the white-hot storage industry to super-nova
– So easy to create virtual machines, and storage is cheap
• But is storage cheap enough?
• Data needs to be virtualized, just like servers
But first, let’s start from the beginning…
34

35. © 2015 Delphix. All Rights Reserved
Are you too busy to Innovate?
Iner'a
35

36. © 2015 Delphix. All Rights Reserved
The Goal: Theory of Constraints
• Long build cycles
are wasteful
• QA/Tes'ng is the
first vicMm
• Improvement not
made at the
constraint is an
illusion
36

37. © 2015 Delphix. All Rights Reserved
The Goal: Theory of Constraints
• Long build cycles
are wasteful
• QA/Tes'ng is the
first vicMm
• Improvement not
made at the
constraint is an
illusion
37

38. © 2015 Delphix. All Rights Reserved
The Goal: Theory of Constraints
• Long build cycles
are wasteful
• QA/Tes'ng is the
first vicMm
• Improvement not
made at the
constraint is an
illusion
38

39. © 2015 Delphix. All Rights Reserved
Factory floor
39

40. © 2015 Delphix. All Rights Reserved
Factory floor
constraint
40

41. © 2015 Delphix. All Rights Reserved
Tune before constraint
constraint
Tuning here
Stock piling
41

42. © 2015 Delphix. All Rights Reserved
Tune a[er constraint
constraint
Tuning here
Starva'on
42

43. © 2015 Delphix. All Rights Reserved
Factory floor: op'mizing at the constraint
constraint
Goal: find constraint and opMmize it
43

44. © 2015 Delphix. All Rights Reserved
Does the Theory of Constraints work for IT ?
• Goals Clarify
• Metrics Define
• Constraints IdenMfy
• PrioriMes Set
• IteraMons Fast
• CI
• Cloud
• Agile
• Kanban
• Kata
“IT is the factory floor of this century” 44

45. © 2015 Delphix. All Rights Reserved
The Phoenix Project: “The Goal” applied to IT
“IT is the factory floor of this century”
What is the
constraint
in IT ?
45

46. © 2015 Delphix. All Rights Reserved
The Phoenix Project: top 5 constraints in IT?
1. QA setup
2. Dev/Build setup
3. Code Architecture
4. Development
5. Product management
- Gene Kim, author of “The Phoenix Project”
“One of the most powerful
things that organizaMons
can do is to enable development
and tesMng to get
environment they need when
they need it“
46

47. © 2015 Delphix. All Rights Reserved
Data is the constraint
60% Projects Over Schedule
85% delayed waiting for data
Data is the Constraint
CIO Magazine Survey:
…only geng worse…
Gartner: Data Doomsday, by 2017 1/3rd IT in crisis
47

48. © 2015 Delphix. All Rights Reserved
• Data Constraint
• AYempted soluMons
• SoluMon
• Use Cases
In this presenta'on :

49. © 2015 Delphix. All Rights Reserved 49
• PROD is what we care most about…
The problem
PROD

50. © 2015 Delphix. All Rights Reserved 50
• PROD is what we care most about…
– But behind every PROD…
• There is QATEST…
The problem
PROD QATEST

51. © 2015 Delphix. All Rights Reserved 51
• PROD is what we care most about…
– But behind every PROD…
• There is QATEST, DEV…
The problem
PROD QATEST UAT

52. © 2015 Delphix. All Rights Reserved 52
• PROD is what we care most about…
– But behind every PROD…
• There is QATEST, DEV…
• …and other project-specific environments to support
The problem
PROD QATEST UAT

53. © 2015 Delphix. All Rights Reserved 53
• Also, as PROD grows…
The problem
PROD PROD

54. © 2015 Delphix. All Rights Reserved 54
UAT QATEST
• Also, as PROD grows…
– Then so do the supporMng non-PROD environments…
The problem
PROD

55. © 2015 Delphix. All Rights Reserved 55
The problem
CEO
COO
CFO CMO CSO CIO
• And so inevitably the situaMon shi^s from this…

56. © 2015 Delphix. All Rights Reserved 56
The problem
CEO
COO
CFO CMO CSO CIO
Storage
Administrator
• To this…
* Inspired by Connor McDonald (hYp://connormcdonald.wordpress.com/)

57. © 2015 Delphix. All Rights Reserved
IT industry is generally unaware of the problem
57

58. © 2015 Delphix. All Rights Reserved
Companies unaware of the problem
Tester or Developer Boss, Storage Admin, DBA
58

59. © 2015 Delphix. All Rights Reserved
• In build…
1. BoYlenecks between
projects
2. Expensive Slow QA/Test
3. WaiMng for environments
4. More Mme spent in setup
than tes'ng
5. ProducMon Bugs
6. Expensive Slow QA
What Problems does the Data Constraint Cause ?
• In operaMons…
1. Storage exhausMon
2. Network saturaMon
3. Admin cycles
4. Unable to fix problems
5. Always behind

60. © 2015 Delphix. All Rights Reserved
Produc'on Wall
60
Produc'on

61. © 2015 Delphix. All Rights Reserved
Life in IT without QA/Test… silos
61

62. © 2015 Delphix. All Rights Reserved
• Data constraint
• AYempted soluMons
• SoluMon
• Use Cases
In this presenta'on:

63. © 2015 Delphix. All Rights Reserved 63
• Some try to miMgate the problem by copying subsets of PROD…
• …but it is difficult and Mme-consuming and expensive to create
referenMally-correct subsets of databases.
Leads to buggy code…
A^emp'ng to solve the problem…
PROD
QATEST DEV UAT

64. © 2015 Delphix. All Rights Reserved 64
• Some try to miMgate the problem by copying subsets of PROD…
• …but it is difficult and Mme-consuming and expensive to create
referenMally-correct subsets of databases.
Leads to buggy code…
A^emp'ng to solve the problem…
PROD
QATEST DEV UAT

65. © 2015 Delphix. All Rights Reserved 65
• Some try to miMgate the problem by copying subsets of PROD…
• …but it is difficult and Mme-consuming and expensive to create
referenMally-correct subsets of databases.
Leads to buggy code…
A^emp'ng to solve the problem…
PROD
QATEST DEV UAT

66. © 2015 Delphix. All Rights Reserved 66
• Cloning producMon to non-product inevitably performed
infrequently…
…meaning that schemas and data are older, dirMer due to
previous tesMng, and too small to accurately represent reality
A^emp'ng to solve the problem…
PROD
QATEST DEV UAT

67. © 2015 Delphix. All Rights Reserved 67
• Cloning producMon to non-product inevitably performed
infrequently…
• …meaning that schemas and data are older, dirMer due to
previous tesMng, and too small to accurately represent reality
A^emp'ng to solve the problem…
PROD
QATEST DEV UAT

68. © 2015 Delphix. All Rights Reserved 68
PROD
• CreaMng snapshots with mirror splits…
– Mirror the volume one extra copy
A^emp'ng to solve the problem…
PROD
PROD

69. © 2015 Delphix. All Rights Reserved 69
PROD
• CreaMng snapshots with mirror splits…
– Mirror the volume one extra copy
– Then cease writes to that extra copy at a specific Mme and make it separately
addressable as a separate volume
A^emp'ng to solve the problem…
PROD
PROD

70. © 2015 Delphix. All Rights Reserved 70
PROD
• CreaMng snapshots with mirror splits…
– Mirror the volume one extra copy
– Then cease writes to that extra copy at a specific Mme and make it separately
addressable as a separate volume
– To create another snapshot, “re-silver” the volume into the mirror set
A^emp'ng to solve the problem…
PROD
PROD

71. © 2015 Delphix. All Rights Reserved 71
PROD
• CreaMng snapshots with mirror splits…
– Mirror the volume one extra copy
– Then cease writes to that extra copy at a specific Mme and make it separately
addressable as a separate volume
– To create another snapshot, “re-silver” the volume into the mirror set
– And it’s ready to split again
A^emp'ng to solve the problem…
PROD
PROD

72. © 2015 Delphix. All Rights Reserved 72
PROD
• CreaMng snapshots with mirror splits…
– Mirror the volume one extra copy
– Then cease writes to that extra copy at a specific Mme and make it separately
addressable as a separate volume
– To create another snapshot, “re-silver” the volume into the mirror set
– And it’s ready to split again
– While we’ve solved the problem of quickly creaMng new copies
A^emp'ng to solve the problem…
PROD
PROD

73. © 2015 Delphix. All Rights Reserved 73
PROD
• CreaMng snapshots with mirror splits…
– Mirror the volume one extra copy
– Then cease writes to that extra copy at a specific Mme and make it separately
addressable as a separate volume
– To create another snapshot, “re-silver” the volume into the mirror set
– And it’s ready to split again
– While we’ve solved the problem of quickly creaMng new copies
• We haven’t solved the problem of using a full image
A^emp'ng to solve the problem…
PROD
PROD

74. © 2015 Delphix. All Rights Reserved 74
PROD
• CreaMng snapshots with mirror splits…
– Mirror the volume one extra copy
– Then cease writes to that extra copy at a specific Mme and make it separately
addressable as a separate volume
– To create another snapshot, “re-silver” the volume into the mirror set
– And it’s ready to split again
– While we’ve solved the problem of quickly creaMng new copies
• We haven’t solved the problem of using a full image
• And re-silvering is not a fast operaMon, so we need Mme to “reload”
A^emp'ng to solve the problem…
PROD
PROD

75. © 2015 Delphix. All Rights Reserved 75
A^emp'ng to solve the problem…
CEO
COO
CFO CMO CSO CIO
Storage
Administrator
We’re sMll at square one…
…and it gets even worse
Because now we have to predict
the size of everything with
perfect accuracy
and then pre-allocate
for copies as well as volumes

76. © 2015 Delphix. All Rights Reserved 76
A^emp'ng to solve the problem…
CEO
COO
CFO CMO CSO CIO
Storage
Administrator
We’re sMll at square one…
…and it gets even worse
Because now we have to predict
the size of everything with
perfect accuracy
and then pre-allocate
for copies as well as volumes

77. © 2015 Delphix. All Rights Reserved 77
A^emp'ng to solve the problem…
CEO
COO
CFO CMO CSO CIO
Storage
Administrator
We’re sMll at square one…
…and it gets even worse
Because now we have to predict
the size of everything with
perfect accuracy
and then pre-allocate
for copies as well as volumes

78. © 2015 Delphix. All Rights Reserved 78
A^emp'ng to solve the problem…
CEO
COO
CFO CMO CSO CIO
Storage
Administrator
We’re sMll at square one…
…and it gets even worse
Because now we have to predict
the size of everything with
perfect accuracy
and then pre-allocate
for copies as well as volumes

79. © 2015 Delphix. All Rights Reserved 79
A^emp'ng to solve the problem…
CEO
COO
CFO CMO CSO CIO
Storage
Administrator
We’re sMll at square one…
…and it gets even worse
Because now we have to predict
the size of everything with
perfect accuracy
and then pre-allocate
for copies as well as volumes

80. © 2015 Delphix. All Rights Reserved 80
So, in a perfect world what we’d like to see is something like this…
A^emp'ng to solve the problem…
PROD
QAT for
Tom
QAT for
Dick
UAT for
Sally
DEV for
Harry
QAT for
Becky

81. © 2015 Delphix. All Rights Reserved 81
A^emp'ng to solve the problem…
PROD
DEV/
QAT/
UAT
Tom Dick Sally Harry Becky
…but, in the real world we see this…

82. © 2015 Delphix. All Rights Reserved 82
• It’s not easy for developers and testers to share environments
simultaneously
– Changes made by each user conflict with one another
• So it becomes necessary to serialize access to the limited
number of environments
– Each user queues at the resource, so their project takes more 'me
elapsed
– Each user destrucMvely changes the resource
• And now we need a way to reset or clean up the environment
between each use
– Reset or clean up takes Mme, which causes further delays between
uses
A^emp'ng to solve the problem…

83. © 2015 Delphix. All Rights Reserved 83
• It’s not easy for developers and testers to share environments
simultaneously
– Changes made by each user conflict with one another
• So it becomes necessary to serialize access to the limited
number of environments
– Each user queues at the resource, so their project takes more 'me
elapsed
– Each user destrucMvely changes the resource
• And now we need a way to reset or clean up the environment
between each use
– Reset or clean up takes Mme, which causes further delays between
uses
A^emp'ng to solve the problem…

84. © 2015 Delphix. All Rights Reserved 84
• It’s not easy for developers and testers to share environments
simultaneously
– Changes made by each user conflict with one another
• So it becomes necessary to serialize access to the limited
number of environments
– Each user queues at the resource, so their project takes more 'me
elapsed
– Each user destrucMvely changes the resource
• And now we need a way to reset or clean up the environment
between each use
– Reset or clean up takes Mme, which causes further delays between
uses
A^emp'ng to solve the problem…

85. © 2015 Delphix. All Rights Reserved 85
• It’s not easy for developers and testers to share environments
simultaneously
– Changes made by each user conflict with one another
• So it becomes necessary to serialize access to the limited
number of environments
– Each user queues at the resource, so their project takes more Mme
elapsed
– Each user destrucMvely changes the resource
• And now we need a way to reset or clean up the environment
between each use
– Reset or clean up takes Mme, which causes further delays between
uses
A^emp'ng to solve the problem…

86. © 2015 Delphix. All Rights Reserved 86
• It’s not easy for developers and testers to share environments
simultaneously
– Changes made by each user conflict with one another
• So it becomes necessary to serialize access to the limited
number of environments
– Each user queues at the resource, so their project takes more Mme
elapsed
– Each user destrucMvely changes the resource
• And now we need a way to reset or clean up the environment
between each use
– Reset or clean up takes Mme, which causes further delays between
uses
A^emp'ng to solve the problem…

87. © 2015 Delphix. All Rights Reserved 87
• It’s not easy for developers and testers to share environments
simultaneously
– Changes made by each user conflict with one another
• So it becomes necessary to serialize access to the limited
number of environments
– Each user queues at the resource, so their project takes more Mme
elapsed
– Each user destrucMvely changes the resource
• And now we need a way to reset or clean up the environment
between each use
– Reset or clean up takes Mme, which causes further delays between
uses
A^emp'ng to solve the problem…

88. © 2015 Delphix. All Rights Reserved 88
• It’s not easy for developers and testers to share environments
simultaneously
– Changes made by each user conflict with one another
• So it becomes necessary to serialize access to the limited
number of environments
– Each user queues at the resource, so their project takes more Mme
elapsed
– Each user destrucMvely changes the resource
• And now we need a way to reset or clean up the environment
between each use
– Reset or clean up takes Mme, which causes further delays between
uses
A^emp'ng to solve the problem…

89. © 2015 Delphix. All Rights Reserved 89
• It’s not easy for developers and testers to share environments
simultaneously
– Changes made by each user conflict with one another
• So it becomes necessary to serialize access to the limited
number of environments
– Each user queues at the resource, so their project takes more 'me
elapsed
– Each user destrucMvely changes the resource
• And now we need a way to reset or clean up the environment
between each use
– Reset or clean up takes Mme, which causes further delays between
uses
A^emp'ng to solve the problem…

90. © 2015 Delphix. All Rights Reserved
• Data constraint
• AYempted soluMons
• SoluMon
• Use Cases
In this presenta'on:

91. © 2015 Delphix. All Rights Reserved 91
• The biggest problem with cloning environments and databases is all
the duplicaMon
Most (if not all) of the blocks comprising the database or file-system exactly
the same
What if we store only one copy of each version of a block?
Copy-on-write (COW)
Never modify a block, but insert a new version instead
Then let the storage figure out what we’re supposed to be seeing?
The storage maintains the relaMonships and versions on retrieval
And then make it portable so that ANY platorm (Unix/Linux, Windows)
can use it?
NFS (network file system)
A solu'on…

92. © 2015 Delphix. All Rights Reserved 92
• The biggest problem with cloning environments and databases is all
the duplicaMon
– Most (if not all) of the blocks comprising the database or file-system
exactly the same
What if we store only one copy of each version of a block?
Copy-on-write (COW)
Never modify a block, but insert a new version instead
Then let the storage figure out what we’re supposed to be seeing?
The storage maintains the relaMonships and versions on retrieval
And then make it portable so that ANY platorm (Unix/Linux, Windows)
can use it?
NFS (network file system)
A solu'on…

93. © 2015 Delphix. All Rights Reserved 93
• The biggest problem with cloning environments and databases is all
the duplicaMon
– Most (if not all) of the blocks comprising the database or file-system
exactly the same
• What if we store only one copy of each version of a block?
Copy-on-write (COW)
Never modify a block, but insert a new version instead
Then let the storage figure out what we’re supposed to be seeing?
The storage maintains the relaMonships and versions on retrieval
And then make it portable so that ANY platorm (Unix/Linux, Windows)
can use it?
NFS (network file system)
A solu'on…

94. © 2015 Delphix. All Rights Reserved 94
• The biggest problem with cloning environments and databases is all
the duplicaMon
– Most (if not all) of the blocks comprising the database or file-system
exactly the same
• What if we store only one copy of each version of a block?
– Copy-on-write (COW)
• Never modify a block, but insert a new version instead
Then let the storage figure out what we’re supposed to be seeing?
The storage maintains the relaMonships and versions on retrieval
And then make it portable so that ANY platorm (Unix/Linux, Windows)
can use it?
NFS (network file system)
A solu'on…

95. © 2015 Delphix. All Rights Reserved 95
• The biggest problem with cloning environments and databases is all
the duplicaMon
– Most (if not all) of the blocks comprising the database or file-system
exactly the same
• What if we store only one copy of each version of a block?
– Copy-on-write (COW)
• Never modify a block, but insert a new version instead
• Then let the storage figure out what we’re supposed to be seeing?
The storage maintains the relaMonships and versions on retrieval
And then make it portable so that ANY platorm (Unix/Linux, Windows)
can use it?
NFS (network file system)
A solu'on…

96. © 2015 Delphix. All Rights Reserved 96
• The biggest problem with cloning environments and databases is all
the duplicaMon
– Most (if not all) of the blocks comprising the database or file-system
exactly the same
• What if we store only one copy of each version of a block?
– Copy-on-write (COW)
• Never modify a block, but insert a new version instead
• Then let the storage figure out what we’re supposed to be seeing?
– The storage maintains the relaMonships and versions on retrieval
And then make it portable so that ANY platorm (Unix/Linux, Windows)
can use it?
NFS (network file system)
A solu'on…

97. © 2015 Delphix. All Rights Reserved 97
• The biggest problem with cloning environments and databases is all
the duplicaMon
– Most (if not all) of the blocks comprising the database or file-system
exactly the same
• What if we store only one copy of each version of a block?
– Copy-on-write (COW)
• Never modify a block, but insert a new version instead
• Then let the storage figure out what we’re supposed to be seeing?
– The storage maintains the relaMonships and versions on retrieval
• And then make it portable so that ANY platorm (Unix/Linux,
Windows) can use it?
NFS (network file system)
A solu'on…

98. © 2015 Delphix. All Rights Reserved 98
• The biggest problem with cloning environments and databases is all
the duplicaMon
– Most (if not all) of the blocks comprising the database or file-system
exactly the same
• What if we store only one copy of each version of a block?
– Copy-on-write (COW)
• Never modify a block, but insert a new version instead
• Then let the storage figure out what we’re supposed to be seeing?
– The storage maintains the relaMonships and versions on retrieval
• And then make it portable so that ANY platorm (Unix/Linux,
Windows) can use it?
– NFS (network file system – Unix/Linux), iSCSI (Windows)
A solu'on…

99. © 2015 Delphix. All Rights Reserved 99
• Cloning the old way…
A solu'on…
PROD QAT

100. © 2015 Delphix. All Rights Reserved 100
• Cloning the smart way…
A solu'on…
PROD QAT
dSource

101. © 2015 Delphix. All Rights Reserved 101
• Cloning the smart way…
– As changes are made to the clone, only those changed blocks are stored…
A solu'on…
QAT
dSource
PROD

102. © 2015 Delphix. All Rights Reserved 102
• Cloning the smart way…
– As changes are made to the clone, only those changed blocks are stored…
• Takes less storage, faster to copy
• The cost of the clone has gone way down in terms of storage and 'me
A solu'on…
QAT
dSource
PROD

103. © 2015 Delphix. All Rights Reserved 103
• Cloning the smart way…
– As changes are made to the clone, only those changed blocks are stored…
• Takes less storage, faster to copy
• The cost of the clone has gone way down in terms of storage and 'me
A solu'on…
CapEx
QAT
dSource
PROD

104. © 2015 Delphix. All Rights Reserved 104
• Cloning the smart way…
– As changes are made to the clone, only those changed blocks are stored…
• Takes less storage, faster to copy
• The cost of the clone has gone way down in terms of storage and 'me
A solu'on…
CapEx OpEx
QAT
dSource
PROD

105. © 2015 Delphix. All Rights Reserved 105
• Overall architecture
• Internal file-system
How does it work?

106. © 2015 Delphix. All Rights Reserved 106
Overall architecture
Delphix
PROD 1
PROD 2
PROD 3
< 50 ms
network
SSH
JDBC
DSP

107. © 2015 Delphix. All Rights Reserved 107
Overall architecture
Delphix
PROD 1
PROD 2
PROD 3
< 50 ms
network
SSH
JDBC
DSP
Not replicaMon (current
point-in-7me), but a
'meflow with
configured retenMon

108. © 2015 Delphix. All Rights Reserved 108
Overall architecture
Delphix
PROD 1
PROD 2
PROD 3
< 50 ms
Virtual machine
• 4, 8, 12, 16 vCPUs
• 32-512 GB vRAM
• < 8-10 ms I/O
latency to
underlying storage
network
SSH
JDBC
DSP
Not replicaMon (current
point-in-7me), but a
'meflow with
configured retenMon

109. © 2015 Delphix. All Rights Reserved 109
VDB 4
VDB 3
VDB 2
Overall architecture
Delphix
PROD 1
PROD 2
PROD 3
VDB 1
network
< 50 ms < 1 ms
Virtual machine
• 4, 8, 12, 16 vCPUs
• 32-512 GB vRAM
• < 8-10 ms I/O
latency to
underlying storage
UNIX/Linux: NFS
Windows: iSCSI
network
SSH
JDBC
DSP
Not replicaMon (current
point-in-7me), but a
'meflow with
configured retenMon

110. © 2015 Delphix. All Rights Reserved
Snapshot 1 – ini'al load
a b c d e f g h i
We start with a full backup. Includes transaction logs needed for recovery.
Jonathan Lewis © 2015
Sn #1
110

111. © 2015 Delphix. All Rights Reserved
Snapshot 2 – incoming new changes
b' c'
a b c d e f g h i
A^er iniMal link/load, Delphix is
incremental forever
Jonathan Lewis © 2015
111
Sn #1

112. © 2015 Delphix. All Rights Reserved
Apply Snapshot 2
a b c d e f g h i b' c'
The Delphix appliance unpacks the backup and "overwrites" the initial
backup with the changed blocks. DxFS makes new copies of the changed
blocks, never updates a block
112
Jonathan Lewis © 2015
Sn #1 Sn #2

113. © 2015 Delphix. All Rights Reserved
Snapshot 3 – more incoming changes
a b c d e f g h i b' c'
113
Jonathan Lewis © 2015
i'
Sn #1 Sn #2

114. © 2015 Delphix. All Rights Reserved
Apply Snapshot 3
a b c d e f g h i b' c'
114
Jonathan Lewis © 2015
i'
Sn #1 Sn #2 Sn #3

115. © 2015 Delphix. All Rights Reserved
Reten'on policies – dropping snapshots
a b c d e f g h i b' c'
115
Jonathan Lewis © 2015
i'
Sn #1 Sn #2 Sn #3

116. © 2015 Delphix. All Rights Reserved
Reten'on policies – dropping snapshots
a d e f g h i b' c'
116
Jonathan Lewis © 2015
i'
Sn #2 Sn #3

117. © 2015 Delphix. All Rights Reserved
Crea'ng a VDB
a d e f g h i b' c'
117
Jonathan Lewis © 2015
i'
Sn #2 Sn #3 VDB #1

118. © 2015 Delphix. All Rights Reserved
Crea'ng a VDB
a d e f g h i b' c'
118
Jonathan Lewis © 2015
i'
Sn #2 Sn #3 VDB #1

119. © 2015 Delphix. All Rights Reserved
Changes to a VDB
a d e f g h i b' c'
119
Jonathan Lewis © 2015
i'
Sn #2 Sn #3 VDB #1
a‘'

120. © 2015 Delphix. All Rights Reserved
Changes to a VDB
a d e f g h i b' c'
120
Jonathan Lewis © 2015
i'
Sn #2 Sn #3 VDB #1
a‘'

121. © 2015 Delphix. All Rights Reserved
OpenZFS features employed within DxFS
• Copy-On-Write transacMon model
• 128-bit file-system
• L1ARC – RAM-based adapMve replacement cache (ARC)
• Dynamic striping
• Lightweight file-system creaMon (snapshots and clones)
• Checksums and Merkle Tree file-system
• Block-level deduplicaMon
• Transparent block compression
• Variable block sizes
• ReplicaMon
• AdapMve endianness
121

122. © 2015 Delphix. All Rights Reserved
Changing reality
122

123. © 2015 Delphix. All Rights Reserved
Changing reality
123

124. © 2015 Delphix. All Rights Reserved
Changing reality
124

125. © 2015 Delphix. All Rights Reserved
Database Virtualiza'on
125

126. © 2015 Delphix. All Rights Reserved
• Data constraint
• AYempted soluMons
• SoluMon
• Use cases
In this presenta'on:

127. © 2015 Delphix. All Rights Reserved
Working with Virtual Data
• Unlimited
• Full size
• Self Service
• Secure
• Cloud onramp
Developing
Tes'ng
Debugging

128. © 2015 Delphix. All Rights Reserved
Working with Virtual Data: Unlimited
GIF by Steve Karam 128

129. © 2015 Delphix. All Rights Reserved
Working with Virtual Data: Full size
129

130. © 2015 Delphix. All Rights Reserved
Working with Virtual Data: Self Service
130

131. © 2015 Delphix. All Rights Reserved
Working with Virtual Data: Secure
131
• Encryp'on reversibly
changes data at rest,
authenMcated and
authorized users may
view
• Redac'on changes
data on retrieval
• Masking irreversibly
changes data at rest
– maintaining relaMonal
integrity across data
distribuMon paYerns

132. © 2015 Delphix. All Rights Reserved
Working with Virtual Data: cloud onramp
132
The cloud is largely a no-fly
zone for proprietary UNIX.
MigraMng from UNIX to
Linux (U2L) is o^en the
first step of a cloud
migraMon.
Data virtualizaMon makes
U2L easier for cross-
platorm development and
tesMng and migraMon

133. © 2015 Delphix. All Rights Reserved
TesMng with Virtual Data
• Fast, parallel
• Rewind
• A/B tesMng
• Sharing

134. © 2015 Delphix. All Rights Reserved
Dev
QA
Instance
Prod
DVA
• Low Resource
• Find bugs Fast
Tes'ng with Virtual Data: Fast, Parallel
Produc'on Time Flow
134

135. © 2015 Delphix. All Rights Reserved
Tes'ng with Virtual Data: Rewind
Instance
QA
Prod
Produc'on Time Flow
135

136. © 2015 Delphix. All Rights Reserved
Tes'ng with Virtual Data: A/B tes'ng
Instance
Instance
Instance
Index 1
Index 2
Produc'on Time Flow
136

137. © 2015 Delphix. All Rights Reserved
Tes'ng with Virtual Data: version control
137
Dev
QA
2.1
Dev
QA
2.2
2.1 2.2
Instance
Prod
DVA Produc'on Time Flow

138. © 2015 Delphix. All Rights Reserved
Tes'ng with Virtual Data: sharing
Instance Jane’s VDB 'meflow
138
Dialogue between QA tester (Jane) and developer (Mary)…
Jane: Mary, I’ve found a data problem in version 3.2.2a.
Mary: Can you bookmark that and share it?
Jane: It is named “Jane 3.2.2a 3pm”, look for dups in tables ABC and XYZ.
Mary: Hmm, I restored it and I see the problem. Thanks!
Instance Mary’s branch
Jane 3.2.2a
3pm

139. © 2015 Delphix. All Rights Reserved
• Projects “12 months to 6 months.”
– New York Life
• Insurance product “about 50 days ... to about 23 days”
– Presbyterian Health
• “Can't imagine working without it”
– State of California
Virtual Data quotes from customers

140. © 2015 Delphix. All Rights Reserved
• Problem: Data is the constraint
• SoluMon: Virtualize the data
• Results
• Half the Mme for projects
• Higher quality, beYer tesMng
• Increase revenue
Summary

141. © 2015 Delphix. All Rights Reserved

142. © 2015 Delphix. All Rights Reserved
Please complete the session evalua'on
We appreciate your feedback and insight
You may complete the session evaluaMon (session #1227) using the mobile app
Email: Tim.Gorman@Delphix.com
Add’l quesMons: hYp://Community.Delphix.com
Blog: hYp://EvDBT.com Mobile: +1 (303) 885-4526
“The Goal: A Process Of Ongoing Improvement” in 1984 by Eliyahu GoldraY
“The Phoenix Project: A Novel About IT, DevOps, And Helping Your Business Win”
in 2012 by Gene Kim, Kevin Behr, George Spafford