You never develop code without version control, why do you develop your database without it? With Liquibase, database changes are stored in human XML-files and committed to the source control system. Changes are applied to the developers local databases. As changes are committed they are distributed to all other environments including all developers local databases, test databases, staging databases, and even to production databases. This presentation will introduce you to Liquibase and the topic database change management. We will also present some advanced topics based on real life experience and a few tips and tricks as well
Rikard Thulin, Squeed and Roger Nilsson, Altran
2. Rikard Thulin
Has been working as a software engineer for 17 years,
mostly as an consultant with focus on the Java platform
e-mail: rikard.thulin@squeed.com, twitter: @rikard_thulin,
blog: http://squeed.com/blog
Roger Nilsson
Has been working as a software engineer and consultant
for 15 years, with focus on the Java platform
e-mail: roger.nilsson@altran.com
About the Presenters
3. Questions?
Questions during the presentation is
encouraged
There will be a Q&A if time permits at
the end of the presentation
4. The Question?
You never develop code without version
control,
why would you develop your
database without it?
5. Liquibase at a glance
Liquibase is an Apache 2.0 Licensed tool for
tracking, managing and applying database
changes
Database changes are stored in an XML (or
JSON, YAML, plain-ish SQL) file that is version
controlled and database independent
Liquibase managed and applies changes to the
database
6. Major Concepts - Changeset
A changeset is uniquely identified change that
should be applied to the database
When Liquibase runs, it queries the
DATABASECHANGELOG table for the
changesets that are marked as executed and
then executes all changesets that have not yet
been executed
8. Major Concepts - Changelog file
The changelog file contains Change Sets or
references to other Changelog files
Changelog files can be be arbitrarily nested for
better management
10. Major Concepts - Changes
Each changeset generally contains a change
which describes the change/refactoring to apply
to the database
Liquibase supports both descriptive changes
that generate SQL for supported databases
and raw SQL
12. Major Concepts - Preconditions
Preconditions can be applied to either the
changelog as a whole or individual change sets
columnExists, tableExists, viewExists, foreignKeyConstraintExists, sqlCheck, and more...
If a precondition fails, Liquibase will stop
execution
13. Major Concepts - Contexts
Contexts can be applied to changesets to
control which are ran in different environments
For example, some changesets can be tagged
as "production" and others as "test"
Use Contexts only when there is a good reason
14. ● Update rollbacks
● Database ”diff“s
● Generating starting change logs from
existing databases
● Generating database change documentation
● Code branches and merging
Liquibase feature set
15. Database Change Management
Database deltas are part of the code change
The correlation between code revisions are
ensured
Allows the automatisation of database changes
Keeps all environments consistent
Allows reverting a previous version of a system
17. Why Liquibase?
8 developers with 8 local databases/schemas
Development server
Test / QA server
CI Server
Staging Server Node 1
Production Node 1
Staging Server Node 2
Production Node 2
18. Why Liquibase?
8 developers with 8 local databases/schemas
Development server
Test / QA server
CI Server
Staging Server Node 1
Production Node 1
Staging Server Node 2
Production Node 2
19. The four steps
1. Write your database change set
2. Run Liquibase locally to test the SQL
3. Commit your source code and change set
4. Deploy application and database changes
Liquibase plays well when merging and working
with branches
20. Structural Refactorings
● Add Column
● Rename Column
● Modify Column
● Drop Column
● Alter Sequence
● Create Table
● Rename Table
● Drop Table
● Create View
● Rename View
● Drop View
● Merge Columns
● Create Stored Procedure
Data Quality Refactorings
● Add Lookup Table
● Add Not-Null Constraint
● Remove Not-Null Constraint
● Add Unique Constraint
● Drop Unique Constraint
● Create Sequence
● Drop Sequence
● Add Auto-Increment
● Add Default Value
● Drop Default Value
Referential Integrity Refactorings
● Add Foreign Key Constraint
● Drop Foreign Key Constraint
● Drop All Foreign Key Constraints
● Add Primary Key Constraint
● Drop Primary Key Constraint
Non-Refactoring Transformations
● Insert Data
● Load Data
● Load Update Data
● Update Data
● Delete Data
● Tag Database
● Stop
Architectural Refactorings
● Create Index
● Drop Index
Custom Refactorings
● Modifying Generated SQL
● Custom SQL
● Custom SQL File
● Custom Refactoring Class
● Execute Shell Command
Supported Refactorings
21. Running Liquibase
● Manual
○ Command Line
○ Ant
○ Maven
● Automatic
○ Spring
○ Servlet Listener
○ CDI Environment
22. Tips and Tricks - real life experience
You must have a manual way to run Liquibase
(command line, ant or maven) during
development of changesets
Bundle your database changes into your
deployment artifact (spring, servlet listener or
CDI)
It is (mentally) scary to do, but it works!
24. Tips and tricks - structure
Think about the structure of your changelogs
Based on experience, when the changelog xml file is
10.000 lines long it is gets ugly. It is much more difficult
when merging and branching.
Have one master changelog that includes smaller
changelogs in turn, divided by release or issue
changelog.xml changelog.xml
jira-1.xml changelog-1_0.xml
jira-4.xml changelog-1_1.xml
25. Tips and tricks - structure
Liquibase should foremost be used for structural changes
but it is also possible to use it for configuration data and for
test data
Keep in mind that the use case for these are very different!
● Structural changes should be applied together with the
corresponding code
● Configuration data has en dependency to the database
structure, but is not applied at any time
● Test data has a dependency to the database structure,
may have preconditions and is typically applied by a CI
server nightly
26. Tips and tricks - structure
● changelog.xml for structural changes, applied when
the system is deployed
● test-data.xml for test data, applied nightly be CI server
● configuration-data.xml for system configuration
changes, applied manually when needed
27. When executing update or rollback there is a
handy option which do not apply changes
against the database but instead save the
resulting SQL
This can be usable when*
● the resulting SQL needs to be tweeked
● having the SQL approved by a DBA
● SOX compliance.
*) Not a recommended best practices by the authors
Advanced Topics
SQL Output
28. Generating RFC to DBA automatically using Jenkins
REQUEST FOR CHANGE for sqlserver://sqlprod
Planning:
1. Changeset esc-1729.xml::1::rikard::(Checksum: 3:be601b760eaccc5d864da5b3639bb031)
Reviewed by: rikard
Tested in sqldev, 2013-06-17 23:45:46, OK | Insert Column, New attribute for customer
Tested in sqltest, 2013-06-18 12:10:01, OK | Insert Column, New attribute for customer
Tested in sqlprod , 2013-09-14 20:57:44, OK | Insert Column, New attribute for customer
Implementation:
Run script using Jenkins deploy server
Implementation & Review:
Executed by Jenkins/CI-Server, 2013-09-14 20:57 Scripts successfully applied
Advanced Topics
SQL Output - real life example
29. Tips and tricks - wrong is right
One of the most difficult challenges is to
overcome “wrong is right” principle
● When a changeset has been committed, it
can not be modified even if it is “wrong”
● You must write another changeset to correct
the previous one
● The “wrong” changeset will hit your
production database followed by the “right”
changeset
30. Tips and tricks - stored procedure
If you have stored procedures the should
obviously be versioned controlled
But the CRC does not fit well as that requires
the stored procedure to be repeated for every
modification
The “run always” attribute saves the day. This
will make sure that whenever there is a change
it is applied -- a better fit for stored procedures
31. Tips and tricks - stored procedures
--liquibase formatted sql
--changeset rikard:1 runOnChange:true
ALTER procedure javaforum
…
● Above we use a plain sql file with two special
comments that magically makes it a
changeset
33. Tips and Tricks - neutralizing data
● The “run always” attribute can also be used
to neutralize data in dev and test
environments
● Useful in cases when the database is copied
for production to other environments
● The changes that has been applied is stored
in the database itself, therefore it is safe to
copy clone a database
34. Tips and Tricks - neutralizing data
<changeSet id="1" author="joe" runAlways="true"
context="sqld,sqlt,localhost">
<comment>Change users psw on (localhost, dev, test) to
'Secret' and their email to 'test@jforum.se'</comment>
<update tableName="user">
<column name="email" value="test@jforum.se"/>
<column name="password" value="Secret"/>
<where>email != 'test@jforum.se' OR password !=
'Secret'</where>
</update>
</changeSet>
35. Real life experience
We have used Liquibase in projects for many years, more
than 50 man years of development with very little problems
Overall score 97%
by Roger and Rikard
