Kubernetes bietet viel Funktionalität, um Zero-Downtime Deployments durchzuführen. Etwas herausfordernder wird es dann, wenn der Service-Update auch mit einem Datenbank-Schema Update verbunden ist. Nebst den verschiedenen Strategien, um ein Datenbankschema in einem Zero-Downtime-Release auszurollen, lernen Sie in diesem Vortrag, wie das Datenbank-Schema sowie die Deployment-Tools in einem Container Verpackt mit der Applikation ausgerollt werden können. Somit erhalten wir ein einziges, in sich konsistentes, Helm Paket, welches den Service samt Datenbank-Schema ausrollen kann.
9. Being ready for 100 deployments a day
Fully automated process
▪ Build Automation
▪ Deployment Automation
▪ Test Automation
Small and frequent releases
▪ Reduce Complexity
▪ Daily Business
There is no place like production
▪ Testing in Production
▪ Zero Downtime
▪ Feature Flags
10. What about the DB development?
Database development
is fully integrated
No manual schema
changes
Automated deployment
of schema changes
14. Database Migrations
New version of the app = different database
schema
Many tools to diff and apply new schemas
Zero-downtime deployments is often a critical
requirement
15. Code First
DB First
Release v2
Deployment Approaches
Deploy
DB Schema
Deploy
Binaries
Prod Environment
v2
Binaries
v1
Prod Environment
v2
Binaries
v2
Release v2
Deploy
Binaries
Deploy
DB Schema
Prod Environment
v1 Binaries
v1
Binaries
v2
Factory
Prod Environment
v2 Binaries
v1
Binaries
v2
Factory
16. Where to put the fallback logic?
Database
▪ Use views / triggers to support old
schema
Advantages
▪ Old code just works during
deployment
Disadvantages
▪ Have a lot of if statement in
database logic
▪ Harder to test
Code
▪ Use factory to determine the
implementation for the current database
version
▪ Couple database version to features /
implementation
Advantages
▪ Code is easier to test
Disadvantages
▪ More complexity in code
▪ Factory / Toggles needed
17. Support Rollback Scenarios
«If you can’t get upgrade right, what leads
you to believe you could get rollback right
as well?” – Buck Hodges
Implement Rollback logic only if needed
▪ DB deployment is often complex and multi-step
▪ Hopefully never used – wasted time for implementation and testing?
18. When to run the migration?
On service startup As part of the deployment
process script
As dedicated jobs within your
application (i.e. k8s jobs)
19. Best Practices
▪ DB Frist deployment mode
▪ Easer to develop
▪ DB Migration is critical – fail fast / don’t deploy binaries
▪ No rollback – forward only
▪ Saves huge effort
▪ PR validation / staging will bring up errors before production deployment
▪ Fully automated process – fast rollout of fixes
▪ Dedicated Deployment Job
▪ Application is self-contained
▪ No dependencies to other deployment scripts
▪ Functionality of target environment
21. CD
PR
Classic CI / CD Pipeline
CI
Checkout
Build
App
Run Unit
Test
Build
Dacpac
Publish
Dacpac
Publish
App
Create DB
Deploy DB
Schema
Deploy
App
QA
Deploy DB
Schema
Deploy
App
Pre-Prod
Clone
Prod DB
Deploy DB
Schema
Deploy
App
Prod
Deploy DB
Schema
Deploy
App
CI Type
25. Current Deployment
Pod #1
v1
Pod #2
v1
Pod #3
v1
Pod #4
v1
Rolling Update 1/4
Pod #1
v1
Pod #2
v1
Pod #3
v1
Pod #4
v1
Rolling Update 2/4
Pod #1
v2
Pod #2
v2
Pod #3
v1
Pod #4
v1
Rolling Update 3/4
Pod #1
v2
Pod #2
v2
Pod #3
v2
Pod #4
v1
Rolling Update 4/4
Pod #1
v2
Pod #2
v2
Pod #3
v2
Pod #4
v2
Rolling Update
26. Migration on service start
Call “db.Database.Migrate” at startup
Problems:
▪ Every instance of the service will attempt to migrate
the database
▪ The application has permissions to perform
destructive updates to the database
27. Migration run by deployment scripts
Use Azure Pipelines / GitHub Actions to run a
deployment script before service rollout
Pro:
▪ Single and dedicated DB deployment
▪ Dedicated security principal for schema deployment
Challenges:
▪ Knowledge in pipeline, application has a dependency
to pipelines to run correctly
28. Using Jobs as part of your application
Use Kubernetes jobs and init containers / Helm chart
hooks
Pro:
▪ Dedicated job with dedicated identity / permissions
▪ Part of target environment, no external
dependencies
Challenges:
▪ More complexity
30. Autonomous Application Packages
▪ CI/CD pipelines work great for internal services
▪ If an application package is distributed, the schema
deployment should be part of it
▪ Logic from the CI/CD pipeline is moved to the
application package
▪ CI/CD pipelines can be simplified
32. Security Considerations
▪ Strict security boundary between dev/test
and prod
▪ Use dedicated users for each database /
service
▪ Use dedicated users for
▪ Schema deployment with DDL
▪ Application / service with read/write permissions
35. Implementing a DB schema
deployment solution
DB schema deployment with Kubernetes releases
36. Create a custom migration runner
▪ Independent (and app specific) tool to
run the DB migration
▪ Developed side-by-side with application
and DB schema
▪ Containerized
▪ Packaged in service deployment
40. Define a Kubernetes Job
▪ Run your DB migration tool as a
Kubernetes job
▪ Use dedicated service identities with
corresponding permissions on database
43. Use init containers
▪ Use init containers to wait for the
migration to successfully finish
▪ Init container will block the deployment /
execution of new application containers
without a successful deployment
▪ Dedicated permissions needed to monitor
jobs
47. Publish single package
▪ Package contains all configurations and
container references to deploy and run
the application
▪ Supports any deployment paradigm /
automated and manual deployment
▪ Ideal solution to distribute your
applications at customer site
48. SQS Server Data Tools
(SSDT)
DB schema deployment with Kubernetes releases
49. SSDT - Characteristic
• SSDT Project Type for relational Database
Development
• Integrated in Visual Studio IDE
• Others: SSMS, Redgate, DDL/DML Scripts
• SSDT Advantages:
IDE
MSBuild
IntelliSense
Validation
Code Base
Consistency
Design
Compare
CI
CD
50. • Officially Supported since VS 2015
• 1:1 Database Representation
• SSDT Deployment / Prerequisites:
SSDT - Characteristic
DB Schema Migrations (Static & Dynamic SQL)
Single Pre- and Post Script Logic
Microsoft.Data.Tools.Msbuild
(NuGet)
51. SSDT - Features
• Build time validation / IntelliSense Support
• Bidirectional Scheme Comparison (SSDT DB)
• Bidirectional Scheme Synchronization (SSDT DB)
• Versioned migration and schemes artifact (DACPAC)
• Code-base integration / Change tracking (GIT)
52. Schema Compare
• Schema Compare
• Local Development
• Bidirectional Sync.
• Choose your
favorite IDE
• Prevent data loss:
rename in SSDT
54. Developer Workflow
1. Create a Feature Branch
from Development
2. Publish/Deploy (F5)
Database Project
3. Develop Database
changes (Renames have
to performed in SSDT)
4. Perform a Schema
Compare from DB to
Database Project, Sync.
5. Commit > PR > Review
Local
DEV DB
Visual Studio
DB Project Git Repo
→
QA Dump
Prod Dump
58. Our Learnings combined…
Features
• Configurable Setup and
Naming Convention
• Logging / Full Transactional
Scripts
• Custom execution filters
• Fully configurable Extension
59. Q & A
DB schema deployment with Kubernetes releases
60. Recap
▪ Dedicated migration runner outside the
service
▪ Use k8s functionality: jobs and init container
▪ Self-contained package, no additional
deployment logic
▪ Database Development fully integrated into
development process
▪ No manual schema changes in deployment
process
61. Thank you for your attention!
If you have any questions do not hesitate to contact us:
4tecture GmbH Marc Müller
Industriestrasse 25 Principal Consultant
CH-8604 Volketswil
+41 44 508 37 00 marc.mueller@4tecture.ch
info@4tecture.ch @muellermarc
www.4tecture.ch www.powerofdevops.com