The webinar discussed challenges with database development and version control. Traditional script-based approaches cause issues like out-of-process changes, working on the wrong code revisions, and lack of history tracking for database changes. The presenters demonstrated a database version control solution that enforces a coordinated development process, integrates with existing version control systems, and provides automated deployment to eliminate errors from manual scripting. Key benefits included parallel development, fast bug detection, and streamlined provisioning with reduced risks from human errors.

1. Thank you for joining us, the webinar will start at:
10:00 Pacific / 11:00 Central / 13:00 East / 18:00 UK Time
Taking Database Development to the 21st Century

2. Presenters
Kyle Hailey @kylehhailey
• Technical Evangelist at Delphix
• Oracle ACE, member of the OakTable Network
Uri Margalit @UriMargalit
• Director, Product Management
• Presenter at world-wide conferences: ODTUG, ilOUG,
Perforce Merge, etc…

3. Before we start
• You will be on mute for the duration of the event
• We are now talking so please type a message
in the Questions box in the Control Panel if you can’t
hear us (please check your speakers and
GoToWebinar audio settings first)
• There will be a Q+A session at the end but please
feel free to type your questions in the Questions
box in the Control Panel in advance
• A recording of the full webinar will be put up online

4. About Delphix
• Founded in 2008, launched in 2010
• CEO Jedidiah Yueh (founder of Avamar: >$1B revenue))
• Based in Silicon Valley, Global Operations
• 20% of Fortune 500

5. About DBmaestro
• Founded in 2008
• Headed by Yariv Tabac and Yaniv Yehuda
• Headquartered in Israel, Global Operations

6. Taking Database Development to
the 21st Century
Kyle Hailey & Uri Margalit

7. Risks
80%
of unplanned
downtime is
due to Change
50%
More than
of this, half is
due to human
errors

8. Technology required

9. Reducing Risk
 Smaller changes Agile
 Automation no human error, more with less
 DevOps synergy

10. Source Control : Code
• GitHub
• SVN
• Perforce
• TFS
• RTC
• VSS
How do you source control the database?

11. Database Challenge
• Crucial
• Must be in sync with application changes
• Objects not file system
– Schema Structure
– PL/SQL Code
– Central resource
• Lacks Version Control
• Relevant data for the task

12. Database Requirements
• How can we
– Enforced version control changes?
– Manage content?
– Branch versions?
– Merge branches?
– Have right data for the task?

13. Data is the constraint
60% Projects Over Schedule
85% delayed waiting for data
Data is the Constraint
CIO Magazine Survey:
only getting worse

14. Why is Data the Constraint ?
1. Development: Bottlenecks & bugs
2. QA: Expensive, single threaded, slow
3. Provisioning: Delays

15. Frustration Waiting
1. Development: Bottlenecks & bugs

16. 1. Development: Bottlenecks & bugs

17. 1. Development: Bottlenecks & bugs: subsets
cause bugs

18. 96% of QA time was building environment
$.04/$1.00 actual testing vs. setup
QA Build QA
QA Build QA
BugX
0
10
20
30
40
50
60
70
1 2 3 4 5 6 7
Delay in Fixing the bug
Cost
To
Correct
Software Engineering Economics – Barry Boehm (1981)
2. QA: slow, expensive, single threaded

19. Management
DBA
System Admin
Storage Admin
Developers
Submit
Request
Disk
Capacity?
Approve
Request $$
(2 Weeks)
Approve
Request $$
(1 Week)
Request
Additional
Storage?
Provision
Capacity
File System
Configured?
Configure LUNS &
Build File System
Coordinate
Replication w/
Infrastructure
Re-
Parameterize &
Configure DB
Mount
Recovery DB
to Specific PIT
Begin
Work
Approve
Request $$
(2 Weeks)
(3 Days)
(3 Days)
(2 Days)
(3 Days)
(3 Days)
…….1-2 Weeks of Approvals, Delays, and Provisioning……
Developer Asks for DB Get Access
Manager approves
DBA Request
system
Setup DB
System
Admin
Request
storage
Setup
machine
Storage
Admin
Allocate
storage
(take snapshot)
3. Delays

20. 3. Delays : culture of no

21. Three Physical Copies Three Virtual Copies

22. Delphix - is software
• Installs as VM
• Any Storage
• Supports
-
-
-
-
- Application Data

23. With Virtual Data
Production
Instance
Database
Dev & QA
Instance
Database
Reporting
Instance
Database
Backup
Instance Instance Instance
Database
InstanceInstance
Database
InstanceInstance
File system
Database

24. What We’ve Seen With Delphix
1. Development: Parallelized, less bugs
2. QA: Low cost, fast bug detection
3. Provisioning: Fast, Culture of Yes

25. Parallel environments
Instance
Instance
Instance
Instance
Source

26. 1. Development : Parallelize
gif by Steve Karam

27. 1. Development: Eliminate bugs

28. 1. Provisioning: Fast, Efficient, Self Service, Culture
of Yes!

29. 1. Development: Version Control
Dev
QA
2.1
Dev
QA
2.2
2.1 2.2
Instance
Prod
DVA

30. Dev
QA
Instance
Prod
DVA
Time Flow
• Low Resource
• Find bugs Fast
2. QA: Fast and Efficient

31. What We’ve Seen With Delphix
1. Development: Parallelized, less bugs
2. QA: Low cost, fast bug detection
3. Provisioning: Fast, Culture of Yes
But …
How do we manage the changes and make sure
the version we deploy is the correct one?

32. Version Control is a constraint
“
“
It was difficult to track who made a change to a
database object and what change they made.
(working-around file based version control)
Sr. DBA @ Large USA Bank
It took hours to get releases working. some changes
were not documented and left out. we actually preferred
crashes in integration. It is much worse when something
works, but works wrong. in production…
(manual and error prone releases)
Sr. R&D Manager @ Credit Card company

33. Version Control is a constraint
We recently had a disaster - the script in the version
control was not updated and when executed in production,
ran the wrong revision. That cost tens of thousands of $
(an out-of-process update to QA that was not properly tracked)
Developer @ Algo Trading company
We had multiple releases to production every day. That
is one release a week with multiple follow up fixes, and yet
more fixes
(code overrides, partial versions, wrong versions – all pushed to production)
CTO @ Credit Card company
“
“

34. Version Control is a constraint
We had an incident where a trigger was not correctly
implemented during a code release. a trigger from a
previous build was used instead which was only detected
on Tuesday morning on the first business day after our code
release. this was a customer-facing application and made
our team look, and feel, bad about the release.
we realized that we needed to bring more discipline and
rigor to our database changes.
(manual process are hard to repeat over and over without errors)
Sr. DBA @ Large USA Bank
“

35. Root Causes
• Manual script based version control process
• Deployment tools unaware of version control
• No red-flags…

36. Two Isolated Processes
Version Control Process
(file based)
Development Process
Check-Out
Script
Modify Script
Get updated
Script from DB
Check-In
Script
Compile
Script
in DB
Debug Script
in DB
?
?
?
?
A
A’

37. Challenges…
Un-coordinated Process
Silos in development
Deployment delays…
Out-of-Process Changes
Errors in production
Lack of confidence in automation
Code overrides
Different methodologies
Lack of history Missing changes
wrong versions

38. Scripts & Version Control
• Challenges…
– Code-overrides
– Working on the wrong revisions
– Scripts do not always find their way to the version control solution
– Out of process updates go unnoticed
– Hard to locate outdated update scripts
• Playing safe? what we really need:
– The actual code of the object
– The upgrade script
– A roll-back script

39. Testing Scripts
• Single object based scripts
– Hard to test in their entirely
– Hard to test due to colliding dependencies
– Need to run in a specific order…
• Large multi object based script
– Represents the entire update - can deal with dependencies
– Much harder to deal with project scope changes
– Hard to mange – a big list of commands

40. 1.11.21.31.41.51.61.7 1.11.11.41.7 1.1.1
X
1.11.1.1
Scripts… Build Once Deploy Many
Int QA Stage Prod
Database Deploy Script
Environment
Re-Base (due to defects)
Dev
Dev
Dev
Model
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
Out of Process
Change
X
X
X
X
X
?
X

41. Scripts are Static…
• Scripts, unless super sophisticated:
– Unaware of changes made in the target environment
– Time passed from their coding to the time they are run
– Potentially overriding production hot-fixes & work done in parallel
by another team
• Content changes are very hard to manage
– Metadata & lookup content does not practically fit into the VC
– In most cases they are simply not managed

42. Poll
What level of management you have for you DB
development?
1. Enforced version control; Baseline aware
automation
2. Version control scripts; Simple automation
3. Version control scripts; Simple Comp & Sync
4. None

43. Dealing With Challenges…
Coordinated ProcessTraceability
Start in the Beginning
No Out-of-Process Changes
Impact Analysis
Automation
Task Based Development
Well Defined Processes

44. Version Control – 1 Enforced Process

45. Dealing With Challenges…
• Integrated Version Control process
– Leverage proven version control best practices
• Forcing check in & out for changes
• Labels
• etc..
– No code-overrides
– Always working with the correct revision
– All changes are documented
• Integrated Version Control process
– Always know who did what, when, why and from where
– No out-of-process changes
– Supporting structure, code and content
• No time spent on manual coding of the change scripts

46. 1.11.21.31.41.51.61.7 1.11.11.41.7 1.1.11.11.1.1
Build & Deploy On Demand
Int QA Stage Prod
Database Deploy Script
Environment
Re-Base (due to defects)
Dev
Dev
Dev
Model
1.1 1.2
1.2 1.3
1.3 1.4
1.4 1.5
1.5 1.6
1.6 1.7
1.1 1.4
1.4 1.7
1.1.1 1.7
Out of Process
Change
1.7
*
1.7

47. Source vs.
Target
Action
= No Action
≠
?
Source vs.
Baseline
Target vs.
Baseline
Action
= = No Action
≠ = Deploy Changes
= ≠ Protect Target
≠ ≠ Merge Changes
You do not have all
of the information
With baselines aware
analysis the unknown is now
known
Simple Compare & Sync Baseline aware Analysis
Deployment Automation - Safety Net
An Index exists in Target (Prod) and not in Source (QA),
what should we do? Drop the Index?

48. Bonus Points – Task Base Development
• Correlate each database change with a change request
– Task ID
– Work Item
– Trouble Ticket
– CR
– etc…
• Partial deployments (a feature, a collection of bugs,
etc…)
• Scope changes easily synced between code and
database

49. Live Demo
• Clone 2 virtual copies of the Trunk
1. Dev1
2. Dev2
• Make changes & merge them into the Trunk:
– Developer1 modifies Dev1
– Developer1 merges changes into the Trunk
– Developer2 modifies Dev2
– Developer2 merges changes into the Trunk
• Rely on enforced changes & automation

50. Data Version Control
Dev1
Instance
Prod
Dev2
Trunk clones
Trunk

51. Dev2
Dev1Merge to dev1
Trunk
Fork
Fork
Fork
Fork

52. Q&A
Kyle Hailey @kylehhailey
Delphix: delphix.com
Uri Margalit @UriMargalit
DBmaestro: dbmaestro.com

53. Thanks!
Kyle Hailey @kylehhailey
Delphix: delphix.com
Uri Margalit @UriMargalit
DBmaestro: dbmaestro.com