Databases create a real challenge for automation and dealing with database deployments is a complex process. Databases contain our most valuable information, business data, which must be preserved and protected at all costs and yet the automation processes for database deployment are not widely adopted.
APNIC Policy Roundup, presented by Sunny Chendi at the 5th ICANN APAC-TWNIC E...
Continuous delivery for databases
1. Continuous Delivery for Databases
SQL Relay – Cardiff - 2014
Presented by: James Smith
Co-Founder, DevOpsGuys
2. Who is this guy?
ABOUT ME
Co-founder @ DevOpsGuys
Two decades building & delivering enterprise web systems
Helped many, many companies implement Continuous Delivery practices
Found High Quality Belgian Beer in late nineties!
3. ABOUT DEVOPSGUYS
We provide
web application management
for customers, giving them access to the expertise necessary for
building, launching, maintaining & optimising applications
allowing them to
accelerate time-to-market
and to
focus on adding value to their business
Who are these guys?
9. DEPLOYMENT RATES
9
300 Deployments / Year
10+ Deployments / Day
50-60 Deployments / Day
Every 11.6 seconds
10. HIGH PERFORMERS
HIGH PERFORMING IT
ORGANISATIONS DEPLOY 30 TIMES
MORE FREQUENTLY, WITH 50%
FEWER FAILURES WITH 8000X
FASTER LEAD TIMES THAN THEIR
PEERS.
10
11. HIGH PERFORMERS
THEY ARE ALSO TWO TIMES MORE LIKELY TO
EXCEED PROFITABILITY, MARKET SHARE &
PRODUCTIVITY GOALS
THEY EXPERIENCE 50% HIGHER MARKET
CAPITALIZATION GROWTH OVER 3 YEARS
11
32. LEGO = DEVOPS
Rapid proto-typing & experimentation
Building blocks – no right or wrong way
Promotes collaboration
Strong cultural appeal
Small batch sizes
Visibly measureable
We’ve even automated it!
Manufacturing
Zenon
Mindstorms
Danish phrase leg godt, which means "play well".
34. != CTRL C, CTRL V
Database deployment is not copying and replacing.
It is the transformation from a previous version to the next version while
preserving data integrity.
Deploying database change is hard
Deploying database change frequently is even harder
37. VERSION CONTROL
Database changes not under VCS
Worse – Changes not “always” committed to VCS
Communication of change
Living in “the sea of ‘branch/merge’ filth”
38. #2: AGAINST THE FLOW
38
Photo Credit: slang589 via Compfight cc
Photo Credit: slang589 via Compfight cc
45. CONWAY'S LAW
"Any organization that designs a system (defined more broadly here than
just information systems) will inevitably produce a design whose structure is
a copy of the organization's communication structure.“
- Conway, 1968
45
46. CONWAY'S LAW
Is (was) a database really needed? 3 Tiers anyone?
Centralised vs Decentralised?
Formalised [change] control
48. #1: GET UNDER CONTROL
Photo Credit: RHiNO NEAL via Compfight cc
49. VERSION CONTROL
It’s code – it should be in VCS!
It’s code – it should be in VCS!
It’s code – it should be in VCS!
Schema & Static/Reference Data
Reverse engineer existing schema & reference data
53. TEST, TEST, TEST
Select the right tests for each stage;
Unit testing
Integration Testing
Deployment Validation
Behaviour Validation
Determine the right data for testing
Do you need it all?
53
60. #6: CHANGE YOUR WAYS
Photo Credit: Stéfan via Compfight cc 60
61. IT’S CULTURE
Technology is only half of the story
Dev’s must work with DBA’s (no silo’s)
Management must think of operations as part of development
Deployment is part of development
Data retention is part of development
Fail faster, but fail safely
61
69. THE TRUTH
Continuous Delivery relies on having 2 basic things;
1. Version Control
2. Automation
70. AND FINALLY…
Customers see results and new features more quickly.
Shorter feedback cycles increases our ability to learn.
Improve the whole system.
Reduce firefighting.
Everyone wins!
70
72. THAT’S ALL FOLKS
Photo Credit: Walter Benson via Compfight cc 72
Editor's Notes
Standard & Poor's 500, is a stock market index based on the market capitalizations of 500 large companies having common stock listed on the NYSE or NASDAQ.
That’s a sign of just how fast computing is changing. But technological change may also be shortening the lifespan of all great companies.
Back in 1958, a company could expect to stay on the list for 61 years. These days, the average is just 18 years.
No one really knows why the rate of turnover is speeding up, but technological disruption could be one big reason. Since 2002, Google, Amazon, and Netflix have joined the S&P 500, while Kodak, the New York Times, Palm and Compaq have all been forced off, essentially by changing technology.
Look at why the companies on the right declined…
They failed to realize the potential of an online revenue stream from a captive customer base
Their technology and IT strategy was compromised by access to enough resource
Customer expectations for service was disrupted by startups using new technology, they didn’t keep up
In all cases, they were too slow to market or too slow to see the opportunity.
He was chairman and CEO of General Electric between 1981 and 2001. During his tenure at GE, the company's value rose 4000%
http://www.stratabridge.com/2012/01/the-growth-control-paradox/rate-of-change-jack-welch/
2001 agile manifesto
Kent Beck published about continuous integration in 1998. CruiseControl was released in 2001.
2010 – Farley & Humble
2012 – DevOps Mainstream
So meet emmet, he’s just deployed some database change.
Data is a risk. Database change is a risk.
DBA group preparing for database deployment
Its not copy and paste (replace) – as per code deployment
Hotfixes made direct to environments
Changes communicated via email, etc
Lots of firefighting from merge conflicts
Dev and Production - Data – Environmental Drift
'Production first' is also used for changes to schemas, reference data, and database logic, leading to 'drift' between the Production database and versions used by development and test teams. – Indexes for performance
Rolls Royce Jet Engine
central database that started out small and gradually grew to become highly unwieldy
applications that depended upon it directly has grown
the presence of all data in the same place was great for rapid development
opaque legacy processes, organisational red-tape, or just inexperience.
#4: Buried under data
Sheer data size can also work against deployability.
Large backups take a long time to run, transmit, and restore, preventing effective use of the database in upstream testing.
In some systesm around 70-80% of the data held in the core transactional database was historical data that was rarely requested or use
moving from 8hours for a backup/transmit/restore cycle to even 30 minutes makes a huge difference in a Continuous Delivery context
Melvin Edward Conway was an early computer scientist, computer programmer, and hacker
For example, if a development team is set up with a SQL database specialist, a JavaScript/CSS developer and a C# developer you they will produce a system with three tiers: a database with stored procedures, a business middle tier and a UI tier. a
Are you getting a database because you have DBA’s
Centralised DBA teams build centralised “MASTER” databases. And vice versa. Team split
tendency to want to “protect the database” from the developers. ITIL, etc?
Typically we see indexes being applied only to non-development systems
Not sure why I used this photo.
Do you know how hard it is to find lego pictures for continuous integration!!
You should be able to assemble your full database from source control *full*
You should be able to patch / upgrade your database (incremental)
3. You should be able to manually test its built
a mix of languages to take advantage of the fact that different languages are suitable for tackling different problems
a relational database where the data is relational, a graph database where the data is highly connected, a document database where the data is more document-like, and so on.
http://martinfowler.com/bliki/PolyglotPersistence.html
Team Collaboration
Audit trail of changes
Rollback to previous version
Automate everything – every step
All changes must be documented and stored in one place